You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@directory.apache.org by Martin Alderson <Ma...@salfordsoftware.co.uk> on 2009/04/02 18:20:52 UTC

RE: [ApacheDS] OutOfMemoryError

> There was a problem in MINA 1.1.7 in 1.5.4. In fact, if you do a big
> search request, responses are stacked into MINA until the searchResultDone
> is found. It has been fixed while moving to MINA 2.0.
> So now, in trunk if you still experience a OOM, then we should investigate
> what happens.
> 
> We also fixed a memory leak somewhere else (don't remember, but Norval
> Hope found it and we applied the patch)
> 
> Can you do some profiling session with Yourkit to see where the memory is
> being sucked ?

I've done some profiling with JMP and found that 1.5.1, 1.5.4 and trunk all seem to be holding on to the SearchResponseEntryImpl objects until the end of the search.


For ApacheDS 1.5.4 and 1.5.1 the SearchResponseEntryImpl objects owner graph looks like:

org.apache.directory.shared.ldap.message.SearchResponseEntryImpl
  owned by org.apache.mina.filter.codec.ProtocolCodecFilter$MessageByteBuffer.message
    owned by org.apache.mina.filter.executor.ExecutorFilter$Event.data
      owned by java.util.LinkedList$Entry.element
        owned by java.util.LinkedList$Entry.(next, previous)

In ApacheDS trunk (as of 31st March):

org.apache.directory.shared.ldap.message.SearchResponseEntryImpl
  owned by org.apache.mina.core.write.DefaultWriteRequest.message
    owned by org.apache.mina.filter.codec.ProtocolCodecFilter$MessageWriteRequest.parentRequest
      owned by Object[1490]
        owned by org.apache.mina.util.CircularQueue.items
          owned by org.apache.mina.core.session.DefaultIoSessionDataStructureFactory$DefaultWriteRequestQueue.q


Strangely ApacheDS 1.5.4 actually streams the search responses to the client immediately - but still keeps the SearchResponseEntryImpl objects in memory.

Any ideas are appreciated!

Thanks,

Martin

Re: [ApacheDS] OutOfMemoryError

Posted by Martin Alderson <eq...@planetquake.com>.
> Ok, so we still have a serious issu we have to fix quickly...
> 
> Can you give us the information about the used JVM, memory used and such 
> so that we can reproduce the problem easilly ?
> 
> May be we should reopen the JIRA too.
> 
> Thanks !

Sun JVM on Windows:
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)

I'm only running 1.5 so I can use JMP though - I have also run tests 
with 1.6 and had the same problem.

I'm starting the server with default settings - which I think is -Xmx 64m.

You should be able to reproduce by just starting with a base ApacheDS 
instance, adding a few thousand entries then searching for them.

I created an LDIF with 25000 users of the form:

dn: uid=user1,ou=users,ou=system
objectclass: person
objectclass: organizationalPerson
objectclass: inetOrgPerson
objectclass: top
cn: user1
displayname: test user
sn: test
uid: test1

then I added them with: ldapmodify -h 127.0.0.1 -p 10389 -D 
uid=admin,ou=system -w secret -f users.ldif -a

I searched with: ldapsearch -h 127.0.0.1 -p 10389 -D uid=admin,ou=system 
-w secret -z 2000 -b ou=users,ou=system -s one -LLL (objectclass=*) 1.1

This has a delay of a few seconds before the 2000 results are shown. 
Searching without the size limit (-z 2000) gives the OutOfMemoryError 
without returning any results.

Martin

Re: [ApacheDS] OutOfMemoryError

Posted by Emmanuel Lecharny <el...@apache.org>.
Martin Alderson wrote:
> Hi Emmanuel,
>
>> There is a BIG bug in MINA 2.0.0-M4 I fixed last week : the 
>> CircularList is not thread safe, and when writing messages into it, 
>> you may have impredictable results. I replaced it with a 
>> ConcurrentList, and the problem is now fixed. We still have to 
>> release MINA 2.0.0-RC1, but I think we will do maybe next week.
>>
>> Can you do that : build MINA trunk, and use it 
>> (MINA-2.0.0-M4-SNAPSHOT) for a new test, to see if the 
>> SearchResponseEntry are still hold ?
>>
>> Thanks !
>
> I've just built mina trunk (producing 
> mina-core-2.0.0-RC1-SNAPSHOT.jar) and tested with that.
>
> Unfortunately it still does the same, i.e. the SearchResponseEntrys 
> are held by the DefaultWriteRequestQueue.
Ok, so we still have a serious issu we have to fix quickly...

Can you give us the information about the used JVM, memory used and such 
so that we can reproduce the problem easilly ?

May be we should reopen the JIRA too.

Thanks !

-- 
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org

Re: [ApacheDS] OutOfMemoryError

Posted by Martin Alderson <eq...@planetquake.com>.
Hi Emmanuel,

> There is a BIG bug in MINA 2.0.0-M4 I fixed last week : the CircularList 
> is not thread safe, and when writing messages into it, you may have 
> impredictable results. I replaced it with a ConcurrentList, and the 
> problem is now fixed. We still have to release MINA 2.0.0-RC1, but I 
> think we will do maybe next week.
> 
> Can you do that : build MINA trunk, and use it (MINA-2.0.0-M4-SNAPSHOT) 
> for a new test, to see if the SearchResponseEntry are still hold ?
> 
> Thanks !

I've just built mina trunk (producing mina-core-2.0.0-RC1-SNAPSHOT.jar) 
and tested with that.

Unfortunately it still does the same, i.e. the SearchResponseEntrys are 
held by the DefaultWriteRequestQueue.

Martin

Re: [ApacheDS] OutOfMemoryError

Posted by Emmanuel Lecharny <el...@apache.org>.
Martin Alderson wrote:
>> There was a problem in MINA 1.1.7 in 1.5.4. In fact, if you do a big
>> search request, responses are stacked into MINA until the searchResultDone
>> is found. It has been fixed while moving to MINA 2.0.
>> So now, in trunk if you still experience a OOM, then we should investigate
>> what happens.
>>
>> We also fixed a memory leak somewhere else (don't remember, but Norval
>> Hope found it and we applied the patch)
>>
>> Can you do some profiling session with Yourkit to see where the memory is
>> being sucked ?
>>     
>
> I've done some profiling with JMP and found that 1.5.1, 1.5.4 and trunk all seem to be holding on to the SearchResponseEntryImpl objects until the end of the search.
>
>
> For ApacheDS 1.5.4 and 1.5.1 the SearchResponseEntryImpl objects owner graph looks like:
>
> org.apache.directory.shared.ldap.message.SearchResponseEntryImpl
>   owned by org.apache.mina.filter.codec.ProtocolCodecFilter$MessageByteBuffer.message
>     owned by org.apache.mina.filter.executor.ExecutorFilter$Event.data
>       owned by java.util.LinkedList$Entry.element
>         owned by java.util.LinkedList$Entry.(next, previous)
>
> In ApacheDS trunk (as of 31st March):
>
> org.apache.directory.shared.ldap.message.SearchResponseEntryImpl
>   owned by org.apache.mina.core.write.DefaultWriteRequest.message
>     owned by org.apache.mina.filter.codec.ProtocolCodecFilter$MessageWriteRequest.parentRequest
>       owned by Object[1490]
>         owned by org.apache.mina.util.CircularQueue.items
>           owned by org.apache.mina.core.session.DefaultIoSessionDataStructureFactory$DefaultWriteRequestQueue.q
>
>
> Strangely ApacheDS 1.5.4 actually streams the search responses to the client immediately - but still keeps the SearchResponseEntryImpl objects in memory.
>
> Any ideas are appreciated!
>   
There is a BIG bug in MINA 2.0.0-M4 I fixed last week : the CircularList 
is not thread safe, and when writing messages into it, you may have 
impredictable results. I replaced it with a ConcurrentList, and the 
problem is now fixed. We still have to release MINA 2.0.0-RC1, but I 
think we will do maybe next week.

Can you do that : build MINA trunk, and use it (MINA-2.0.0-M4-SNAPSHOT) 
for a new test, to see if the SearchResponseEntry are still hold ?

Thanks !

-- 
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org