You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@directory.apache.org by Emmanuel Lécharny <el...@gmail.com> on 2018/10/13 01:43:48 UTC
Re: apacheDS JVM process crashes with OutOfMemory
Le 12/10/2018 à 22:14, Pankaj Rathod a écrit :
> Hi Emmanuel,
>
> I could finally generate dumps in hprof format. I have shared you heapdumps
> (java.2018.10.01-11.07.22.hprof) in same folder where earlier I had sent
> PHD format dumps.
>
> This heapdump is for very small heap memory size. But you can notice in
> this dump that there are around 1818 objects of NioSocketSession class.
>
> In our system environment (and in our PHD dumps), we see around 80000
> objects of NioSocketSession class, which combinely occupying around 95% of
> heap memory. This is the leak suspect as per our PHD dump analysis. The
> heap analyzer tool mentions One instance of NioSocketAcceptor occupies 95%
> memory.
>
> Please ignore the memory occupied by ACLTableGenerator class in heapdumps.
> It has fixed size. It is a custom class developed for our need.
Hi !
I just quickly checked the provided hprof file, which has 2469
NioSession instances.
What it means is that many LDAP conections have been initiated, and
never dropped.
At this point, I can see 2 scenario :
- the clients start a connection, and never close it
- the clients start a connection, close it, but the server never ditch it
In the second case, we should check wht happens when the client close
the connection. The NioSession object should be discarded. The first
scenario is more annoying : it's perfectly possible that the client
never close the connection, so the serve rkeep the NioSession forever.
We should have a timeout for this case, which automatically closes the
connection after a configured innactivity timeout.
In your test, can you tell me if your client is closing the connection,
and how ? Thanks !
--
Emmanuel Lecharny
Symas.com
directory.apache.org
Re: apacheDS JVM process crashes with OutOfMemory
Posted by Pankaj Rathod <pa...@gmail.com>.
Hi Emmanuel,
>> Question : would it be possible for you to share your load test, so that
I can reproduc it on my server to see what's going on ?
There is no specific test case. We are observing the issue with any search
filter.
Perform below steps 10 times in a second
Step1: SimpleBind to SSL connection
Step2: Search
Step3: Unbind
After 3-4 hours run of load test, I see multiple number of NioSocketSession
in apacheDS heap dumps.
I have shared you java project here which performs the load test. It will
be really helpful if you can reproduce the issue on your apacheDS instance
(M23 version only).
Thanks and Regards,
Pankaj Rathod
On Tue, Oct 16, 2018 at 10:34 AM Emmanuel Lécharny <el...@gmail.com>
wrote:
> Hi Pankaj,
>
> Le 16/10/2018 à 16:56, Pankaj Rathod a écrit :
> > Hi Emmanuel,
> >
> > Thanks for analyzing the heapdumps.
> >
> > You have mentioned two possible scenarios for this accumulation of
> > NioSocketSession objects.
> >
> > 1. - the clients start a connection, and never close it.
> > After finishing the testrun, I had checked number of connections to port
> > using netstat command. There were 0 connections. So I don't think this
> can
> > be the cause.
> > I checked below workitem is mentioned in apacheDS jira. Is this going to
> be
> > included in any upoming apacheDS release?
> >
> http://mail-archives.apache.org/mod_mbox/directory-dev/201808.mbox/%3CJIRA.13144434.1520885847000.120166.1534807140215@Atlassian.JIRA%3E
>
> I don't think so.
>
> The probem with this approach is that it would eat a lot of CPU checking
> sessions every second, for a very little gain:
> - most of the sessions will always be properly closed: they will be
> opened, do some operation, and then closed. If there is some delay
> between the opening and the closing, we will check every second for
> their idle status, for nothing if the client is properly written
> - if you have tens of thousands sessions, that will litterally kill the
> server as every second, it will check each session for its status.
>
> I'd rather depend on the underlying OS to properly handle TCP connections.
>
> >
> > 2. - the clients start a connection, close it, but the server never ditch
> > it. In your test, can you tell me if your client is closing the
> connection,
> > and how ?
> > Our testrun client executes new threads repetitively where every thread
> > performs below operations before terminating the thread.
> > execute bind
> > perform search
> > execute unbind
>
> Sounds fine.
>
> > We are observing memory growth if we enable or disable the
> "execute
> > unbind" task for the threads before termination.
>
> Sounds like a bug.
>
> >
> > We did some investigation in MINA for our test run and observed that the
> > session objects present in memory for longer time have status "Closing".
> We
> > are expecting this could be because of below issue
> > https://issues.apache.org/jira/browse/DIRMINA-1021
>
> Unlikely. This JIRA has been fixed in MINA 2.0.14, ApacheDS M24 uses
> MINA 2.0.16
>
> >
> > The fix for this issue seems included in MINA version which is bundled in
> > apacheDS M24. Going to apacheDS M24/M25 will be time consuming process
> for
> > us. Although, we will plan to upgrade in future.
>
> Switching to ApacheDS M25 should not take that much time: you just have
> to reload the database from scratch, as the format has changed.
>
> >
> > Emmanuel, could you comment on below queries related to MINA issues for
> > apacheDS? Your response will help us deciding to test the fix for
> > DIRMINA-1021. If you suggest then I will forward this mailthread on MINA
> > forum instead.
> > 1. Can heapmemory get full because of this issue? Or something in
> > MINA/apacheDS will remove these accumulated CLOSING state
> NioSocketSession
> > objects? Will the heapdumps show the exact same picture (top consumer
> > hierarchy) if memory crashes because of this issue? We noticed in our
> > heapdumps that top consumer hierarchy is almost similar to the hprof I
> > shared with you. We observed large number of NioSocketSession objects
> were
> > occupying more than 95% of memory.
>
> I don't think MINA is the culprit here. There is clearly some
> investigation to be done to see if we are correctly handling closing
> session in ApacheDS.
>
> > 2. The mailthread for DIRMINA-1021 mentions code changes as
> > "writebuffer method should call removeNow method when IOException
> occurs".
> > Could you please comment if this single line code change needed for this
> > fix? (If you suggest, then I will forward this question on MINA forum
> > instead.)
>
> I don't reming from the top of my head what was the exact fix was, it
> was 3 years ago, more or less :-)
>
>
> Question : would it be possible for you to share your load test, so that
> I can reproduc it on my server to see what's going on ?
>
> Thanks !
>
> --
> Emmanuel Lecharny
>
> Symas.com
> directory.apache.org
>
>
Re: apacheDS JVM process crashes with OutOfMemory
Posted by Emmanuel Lécharny <el...@gmail.com>.
Hi Pankaj,
Le 16/10/2018 à 16:56, Pankaj Rathod a écrit :
> Hi Emmanuel,
>
> Thanks for analyzing the heapdumps.
>
> You have mentioned two possible scenarios for this accumulation of
> NioSocketSession objects.
>
> 1. - the clients start a connection, and never close it.
> After finishing the testrun, I had checked number of connections to port
> using netstat command. There were 0 connections. So I don't think this can
> be the cause.
> I checked below workitem is mentioned in apacheDS jira. Is this going to be
> included in any upoming apacheDS release?
> http://mail-archives.apache.org/mod_mbox/directory-dev/201808.mbox/%3CJIRA.13144434.1520885847000.120166.1534807140215@Atlassian.JIRA%3E
I don't think so.
The probem with this approach is that it would eat a lot of CPU checking
sessions every second, for a very little gain:
- most of the sessions will always be properly closed: they will be
opened, do some operation, and then closed. If there is some delay
between the opening and the closing, we will check every second for
their idle status, for nothing if the client is properly written
- if you have tens of thousands sessions, that will litterally kill the
server as every second, it will check each session for its status.
I'd rather depend on the underlying OS to properly handle TCP connections.
>
> 2. - the clients start a connection, close it, but the server never ditch
> it. In your test, can you tell me if your client is closing the connection,
> and how ?
> Our testrun client executes new threads repetitively where every thread
> performs below operations before terminating the thread.
> execute bind
> perform search
> execute unbind
Sounds fine.
> We are observing memory growth if we enable or disable the "execute
> unbind" task for the threads before termination.
Sounds like a bug.
>
> We did some investigation in MINA for our test run and observed that the
> session objects present in memory for longer time have status "Closing". We
> are expecting this could be because of below issue
> https://issues.apache.org/jira/browse/DIRMINA-1021
Unlikely. This JIRA has been fixed in MINA 2.0.14, ApacheDS M24 uses
MINA 2.0.16
>
> The fix for this issue seems included in MINA version which is bundled in
> apacheDS M24. Going to apacheDS M24/M25 will be time consuming process for
> us. Although, we will plan to upgrade in future.
Switching to ApacheDS M25 should not take that much time: you just have
to reload the database from scratch, as the format has changed.
>
> Emmanuel, could you comment on below queries related to MINA issues for
> apacheDS? Your response will help us deciding to test the fix for
> DIRMINA-1021. If you suggest then I will forward this mailthread on MINA
> forum instead.
> 1. Can heapmemory get full because of this issue? Or something in
> MINA/apacheDS will remove these accumulated CLOSING state NioSocketSession
> objects? Will the heapdumps show the exact same picture (top consumer
> hierarchy) if memory crashes because of this issue? We noticed in our
> heapdumps that top consumer hierarchy is almost similar to the hprof I
> shared with you. We observed large number of NioSocketSession objects were
> occupying more than 95% of memory.
I don't think MINA is the culprit here. There is clearly some
investigation to be done to see if we are correctly handling closing
session in ApacheDS.
> 2. The mailthread for DIRMINA-1021 mentions code changes as
> "writebuffer method should call removeNow method when IOException occurs".
> Could you please comment if this single line code change needed for this
> fix? (If you suggest, then I will forward this question on MINA forum
> instead.)
I don't reming from the top of my head what was the exact fix was, it
was 3 years ago, more or less :-)
Question : would it be possible for you to share your load test, so that
I can reproduc it on my server to see what's going on ?
Thanks !
--
Emmanuel Lecharny
Symas.com
directory.apache.org
Re: apacheDS JVM process crashes with OutOfMemory
Posted by Pankaj Rathod <pa...@gmail.com>.
Hi Emmanuel,
Thanks for analyzing the heapdumps.
You have mentioned two possible scenarios for this accumulation of
NioSocketSession objects.
1. - the clients start a connection, and never close it.
After finishing the testrun, I had checked number of connections to port
using netstat command. There were 0 connections. So I don't think this can
be the cause.
I checked below workitem is mentioned in apacheDS jira. Is this going to be
included in any upoming apacheDS release?
http://mail-archives.apache.org/mod_mbox/directory-dev/201808.mbox/%3CJIRA.13144434.1520885847000.120166.1534807140215@Atlassian.JIRA%3E
2. - the clients start a connection, close it, but the server never ditch
it. In your test, can you tell me if your client is closing the connection,
and how ?
Our testrun client executes new threads repetitively where every thread
performs below operations before terminating the thread.
execute bind
perform search
execute unbind
We are observing memory growth if we enable or disable the "execute
unbind" task for the threads before termination.
We did some investigation in MINA for our test run and observed that the
session objects present in memory for longer time have status "Closing". We
are expecting this could be because of below issue
https://issues.apache.org/jira/browse/DIRMINA-1021
The fix for this issue seems included in MINA version which is bundled in
apacheDS M24. Going to apacheDS M24/M25 will be time consuming process for
us. Although, we will plan to upgrade in future.
Emmanuel, could you comment on below queries related to MINA issues for
apacheDS? Your response will help us deciding to test the fix for
DIRMINA-1021. If you suggest then I will forward this mailthread on MINA
forum instead.
1. Can heapmemory get full because of this issue? Or something in
MINA/apacheDS will remove these accumulated CLOSING state NioSocketSession
objects? Will the heapdumps show the exact same picture (top consumer
hierarchy) if memory crashes because of this issue? We noticed in our
heapdumps that top consumer hierarchy is almost similar to the hprof I
shared with you. We observed large number of NioSocketSession objects were
occupying more than 95% of memory.
2. The mailthread for DIRMINA-1021 mentions code changes as
"writebuffer method should call removeNow method when IOException occurs".
Could you please comment if this single line code change needed for this
fix? (If you suggest, then I will forward this question on MINA forum
instead.)
Thanks and Regards,
Pankaj Rathod
On Fri, Oct 12, 2018 at 8:43 PM Emmanuel Lécharny <el...@gmail.com>
wrote:
>
>
> Le 12/10/2018 à 22:14, Pankaj Rathod a écrit :
> > Hi Emmanuel,
> >
> > I could finally generate dumps in hprof format. I have shared you
> heapdumps
> > (java.2018.10.01-11.07.22.hprof) in same folder where earlier I had sent
> > PHD format dumps.
> >
> > This heapdump is for very small heap memory size. But you can notice in
> > this dump that there are around 1818 objects of NioSocketSession class.
> >
> > In our system environment (and in our PHD dumps), we see around 80000
> > objects of NioSocketSession class, which combinely occupying around 95%
> of
> > heap memory. This is the leak suspect as per our PHD dump analysis. The
> > heap analyzer tool mentions One instance of NioSocketAcceptor occupies
> 95%
> > memory.
> >
> > Please ignore the memory occupied by ACLTableGenerator class in
> heapdumps.
> > It has fixed size. It is a custom class developed for our need.
>
> Hi !
>
> I just quickly checked the provided hprof file, which has 2469
> NioSession instances.
>
> What it means is that many LDAP conections have been initiated, and
> never dropped.
>
> At this point, I can see 2 scenario :
>
> - the clients start a connection, and never close it
> - the clients start a connection, close it, but the server never ditch it
>
> In the second case, we should check wht happens when the client close
> the connection. The NioSession object should be discarded. The first
> scenario is more annoying : it's perfectly possible that the client
> never close the connection, so the serve rkeep the NioSession forever.
> We should have a timeout for this case, which automatically closes the
> connection after a configured innactivity timeout.
>
> In your test, can you tell me if your client is closing the connection,
> and how ? Thanks !
>
> --
> Emmanuel Lecharny
>
> Symas.com
> directory.apache.org
>
>
Re: apacheDS JVM process crashes with OutOfMemory
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 15/10/2018 à 10:05, Lothar Haeger a écrit :
> Emmanuel Lécharny wrote:
>
>> - the clients start a connection, and never close it
>
> "Smart" firewalls and VPNs often drop connections they consider "idle" without
> notifying client nor server. Definitely a case to handle on both sides, since
> there's little hope to fix all existing firewalls and all firewall developer's
> brains, ever.
If a VPN or a FW drops some connections, those connections will be
closed when the TCP timeout will be reached (depending on teh OS
configuration). This is often set to 7200s on a linux box, ie 2h.
If you conduct a load test where the client does not properly close the
connection, you are likely to have quickly thousands of pending
NioSession, which will be closed when teh timeout is reached. That also
mean you have to be ready for such a scenario, and that means pouring
memory on your server...
That being said, there is a mechanism that could be used to dectect
quickly if a session is not anymore valid: we can check its idling
status. There are at least 3 problems with this approach :
- Persistent Search and Replication connection shud bypass it
- if the idling timeout is set too low, you could perfeclty disconnect
session that are supposed to be valid
- it's CPU expensive, as we check each session for its idling status
every second. If you have thousands of them, then it's quickly becoming
really CPU consuming...
--
Emmanuel Lecharny
Symas.com
directory.apache.org
Re: apacheDS JVM process crashes with OutOfMemory
Posted by Lothar Haeger <lo...@brummelhook.com>.
Emmanuel Lécharny wrote:
> - the clients start a connection, and never close it
"Smart" firewalls and VPNs often drop connections they consider "idle" without
notifying client nor server. Definitely a case to handle on both sides, since
there's little hope to fix all existing firewalls and all firewall developer's
brains, ever.