You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ode.apache.org by Stan Angeloff <st...@angeloff.name> on 2014/07/14 12:45:23 UTC
Possible memory leak for long-running continuous instances?
Hello,
I've been playing with ODE for quite a while now and one of my most recent
attempts tried to run 5, 000 simultaneous instances of one process.
Unfortunately after a while ODE uses up all of the allowed memory and barfs
'Java heap space' continuously in the log.
My set up is very basic, starting from
https://github.com/vanto/apache-ode-tomcat-bundle (which has been an
excellent learning tool). The main .bpel file consists of a continuous
Repeat Until inside of which there are two sequential invoke activities
calling external SOAP services.
When the 5, 000 instances are started, the memory footprint is fairly
stable. After a while, though, org.apache.axis2.engine.AxisConfiguration
accumulates a lot of org.apache.axis2.description.AxisService instances
(all of which have their name property set to 'anonService' + hash). Those
keep growing until eventually the server stops responding due to exhausted
memory. I have taken heap dumps at various stages and examined those.
Eclipse was even smart enough to detect the potential memory leak on the
latest dump just as the server crashed.
One instance of "org.apache.axis2.engine.AxisConfiguration" loaded by
> "org.apache.catalina.loader.WebappClassLoader @ 0xe07e53e0" occupies
> 390,800,064 (75.91%) bytes. The memory is accumulated in one instance of
> "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "<system class
> loader>".
I'll be happy to export the complete project, note however it doesn't
matter what services are invoked or what the .bpel does so long it's a)
continuous, e.g., does not stop b) calls an external service.
I tried looking into ODE/Axis2 to try and figure out what might be going
wrong. My understanding is that service clients are cached and recycled
in SoapExternalService.java (
https://github.com/apache/ode/blob/master/axis2/src/main/java/org/apache/ode/axis2/SoapExternalService.java#L265)
When a new ServiceClient instance is created, it calls the
configureServiceClient method within Axis2 with axisService = null (
https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L143).
That in turn calls createAnonymousService. Later on, ODE calls
setAxisService which forces Axis2 to dispose of the anonymous service it
created earlier.
If the above is indeed what is happening, then there should be no memory
leaks. However, in the memory dumps the number of instances of
org.apache.axis2.description.AxisService keeps growing. I've tried
increasing the maximum heap size up to 8GB and it will fail again, it is
just a matter of time.
Has anyone else experienced this? Is anyone interested in trying to help me
narrow down the cause of this? Is it possible it's not even a memory issue?
Regards,
Stan
Re: Possible memory leak for long-running continuous instances?
Posted by Stan Angeloff <st...@angeloff.name>.
Hi Sathwik,
Try wrapping the process in a Repeat Until activity with an always false
condition (e.g., true() = false() ). I usually put a <wait><for /></wait>
so that the service gets called about every 30 seconds (back-off). Start 1,
000 instances of that process (effectively making 2, 000 req/min) and let
them run for a while. The memory will soon start leaking.
ODE already has a workaround for this particular memory issue on process
termination. The key is to make the process run for a long time and make
lots of calls to the same service from within the lifetime of one process.
I'll work on extracting the relevant project artifacts and will send
separately as a ZIP.
Thanks for looking into this.
On 25 July 2014 08:24, Sathwik B P <sa...@gmail.com> wrote:
> Hi Stan,
>
> Would like your help in replicating this issue.
>
> I have devised a process which calls an external service. I have a
> continuous looping call to the process from the soapUI.
> I have run about 4000 instances a couple of times, and I don't see
> AxisService Instances going beyond 110 as my threads per host is equal to
> 100.
>
> regards,
> sathwik
>
>
>
>
> On Tue, Jul 15, 2014 at 12:59 PM, Sathwik B P <sa...@gmail.com>
> wrote:
>
> > Hi Stan,
> >
> > Thanks for reporting and your patch.
> >
> > Will look into it when I get some freetime.
> >
> > For your question about building ODE with axis2 1.6, it's not possible at
> > the moment as it needs some changes in the Axiom usage within ODE.
> >
> > Will see if we can bring it on 1.3.7 release, though the timelines are
> not
> > fixed yet.
> >
> > regards,
> > sathwik
> >
> >
> > On Mon, Jul 14, 2014 at 10:15 PM, Stan Angeloff <st...@angeloff.name>
> > wrote:
> >
> >> For those interested, I ended up patching Axis2 1.5.6 from source with
> >>
> >>
> https://gist.github.com/StanAngeloff/e245a1a349d04efcfad6#file-axis2-1-5-6-cleanup-patch
> >> The patch has been adapted from two commits on the 1_6 branch. I
> >> re-compiled the kernel module and swapped the one that comes with the
> >> binary distribution of ODE. This seems to have resolved the issue for
> me.
> >>
> >> I also found https://issues.apache.org/jira/browse/ODE-585, however
> it's
> >> tagged for 2.0. IMHO memory leaks are a pretty major issue and should
> get
> >> addressed in whatever is the current stable release?
> >>
> >>
> >> On 14 July 2014 14:31, Stan Angeloff <st...@angeloff.name> wrote:
> >>
> >> > *tl;dr;* Is it possible to use Axis 1.6 in ODE? How does one go about
> >> > building ODE and bundling it with Axis 1.6?
> >> >
> >> > Digging a bit deeper in the log files with
> >> > log4j.category.org.apache.axis2=DEBUG and
> >> > log4j.category.org.apache.ode=DEBUG configured yields interesting
> >> results.
> >> > When a ServiceClient instance is created, the second parameter is
> NULL.
> >> > When this is the case, Axis2 will create a new anonymous service (
> >> >
> >>
> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L137
> >> ).
> >> > The anonymous service is then added to the AxisConfiguration instance
> (
> >> >
> >>
> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L176
> >> > ).
> >> >
> >> > Internally, this creates a new AxisServiceGroup (
> >> >
> >>
> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L315
> >> ).
> >> > This then adds the new anonymous service to an allEndpoints property (
> >> >
> >>
> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L373
> >> > ).
> >> >
> >> > When ODE calls setAxisService, Axis2 clean up the service (
> >> >
> >>
> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L612
> >> ).
> >> > It does not, however, clean up the reference from allEndpoints.
> >> >
> >> > The filtered catalina.out
> >> > <
> >>
> https://gist.githubusercontent.com/StanAngeloff/9f1798e34963012bc190/raw/catalina.out.filtered
> >> >
> >> > log is attached for reference. Analysing the heap dumps indicates that
> >> > allEndpoints is indeed the culprit.
> >> >
> >> > Axis 1.6 seems to have a fix for this issue:
> >> >
> >> >
> >>
> https://github.com/apache/axis2-java/blob/1_6/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L679
> >> >
> >> > Is it possible to use Axis 1.6 in ODE? How does one go about building
> >> ODE
> >> > and bundling it with Axis 1.6?
> >> > Should the ODE 1.3.x branch be updated to use the latest stable Axis2?
> >> >
> >> > Regards,
> >> > Stan
> >> >
> >> >
> >> >
> >> >
> >> > On 14 July 2014 13:45, Stan Angeloff <st...@angeloff.name> wrote:
> >> >
> >> >> Hello,
> >> >>
> >> >> I've been playing with ODE for quite a while now and one of my most
> >> >> recent attempts tried to run 5, 000 simultaneous instances of one
> >> process.
> >> >> Unfortunately after a while ODE uses up all of the allowed memory and
> >> barfs
> >> >> 'Java heap space' continuously in the log.
> >> >>
> >> >> My set up is very basic, starting from
> >> >> https://github.com/vanto/apache-ode-tomcat-bundle (which has been an
> >> >> excellent learning tool). The main .bpel file consists of a
> continuous
> >> >> Repeat Until inside of which there are two sequential invoke
> activities
> >> >> calling external SOAP services.
> >> >>
> >> >> When the 5, 000 instances are started, the memory footprint is fairly
> >> >> stable. After a while, though,
> >> org.apache.axis2.engine.AxisConfiguration
> >> >> accumulates a lot of org.apache.axis2.description.AxisService
> instances
> >> >> (all of which have their name property set to 'anonService' + hash).
> >> >> Those keep growing until eventually the server stops responding due
> to
> >> >> exhausted memory. I have taken heap dumps at various stages and
> >> examined
> >> >> those. Eclipse was even smart enough to detect the potential memory
> >> leak on
> >> >> the latest dump just as the server crashed.
> >> >>
> >> >> One instance of "org.apache.axis2.engine.AxisConfiguration" loaded by
> >> >>> "org.apache.catalina.loader.WebappClassLoader @ 0xe07e53e0" occupies
> >> >>> 390,800,064 (75.91%) bytes. The memory is accumulated in one
> instance
> >> of
> >> >>> "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by
> "<system
> >> class
> >> >>> loader>".
> >> >>
> >> >>
> >> >> I'll be happy to export the complete project, note however it doesn't
> >> >> matter what services are invoked or what the .bpel does so long it's
> a)
> >> >> continuous, e.g., does not stop b) calls an external service.
> >> >>
> >> >> I tried looking into ODE/Axis2 to try and figure out what might be
> >> going
> >> >> wrong. My understanding is that service clients are cached and
> recycled
> >> >> in SoapExternalService.java (
> >> >>
> >>
> https://github.com/apache/ode/blob/master/axis2/src/main/java/org/apache/ode/axis2/SoapExternalService.java#L265
> >> )
> >> >> When a new ServiceClient instance is created, it calls the
> >> >> configureServiceClient method within Axis2 with axisService = null (
> >> >>
> >>
> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L143
> >> ).
> >> >> That in turn calls createAnonymousService. Later on, ODE calls
> >> >> setAxisService which forces Axis2 to dispose of the anonymous service
> >> it
> >> >> created earlier.
> >> >>
> >> >> If the above is indeed what is happening, then there should be no
> >> memory
> >> >> leaks. However, in the memory dumps the number of instances of
> >> >> org.apache.axis2.description.AxisService keeps growing. I've tried
> >> >> increasing the maximum heap size up to 8GB and it will fail again, it
> >> is
> >> >> just a matter of time.
> >> >>
> >> >> Has anyone else experienced this? Is anyone interested in trying to
> >> help
> >> >> me narrow down the cause of this? Is it possible it's not even a
> memory
> >> >> issue?
> >> >>
> >> >> Regards,
> >> >> Stan
> >> >>
> >> >
> >> >
> >>
> >
> >
>
Re: Possible memory leak for long-running continuous instances?
Posted by Sathwik B P <sa...@gmail.com>.
Hi Stan,
Would like your help in replicating this issue.
I have devised a process which calls an external service. I have a
continuous looping call to the process from the soapUI.
I have run about 4000 instances a couple of times, and I don't see
AxisService Instances going beyond 110 as my threads per host is equal to
100.
regards,
sathwik
On Tue, Jul 15, 2014 at 12:59 PM, Sathwik B P <sa...@gmail.com> wrote:
> Hi Stan,
>
> Thanks for reporting and your patch.
>
> Will look into it when I get some freetime.
>
> For your question about building ODE with axis2 1.6, it's not possible at
> the moment as it needs some changes in the Axiom usage within ODE.
>
> Will see if we can bring it on 1.3.7 release, though the timelines are not
> fixed yet.
>
> regards,
> sathwik
>
>
> On Mon, Jul 14, 2014 at 10:15 PM, Stan Angeloff <st...@angeloff.name>
> wrote:
>
>> For those interested, I ended up patching Axis2 1.5.6 from source with
>>
>> https://gist.github.com/StanAngeloff/e245a1a349d04efcfad6#file-axis2-1-5-6-cleanup-patch
>> The patch has been adapted from two commits on the 1_6 branch. I
>> re-compiled the kernel module and swapped the one that comes with the
>> binary distribution of ODE. This seems to have resolved the issue for me.
>>
>> I also found https://issues.apache.org/jira/browse/ODE-585, however it's
>> tagged for 2.0. IMHO memory leaks are a pretty major issue and should get
>> addressed in whatever is the current stable release?
>>
>>
>> On 14 July 2014 14:31, Stan Angeloff <st...@angeloff.name> wrote:
>>
>> > *tl;dr;* Is it possible to use Axis 1.6 in ODE? How does one go about
>> > building ODE and bundling it with Axis 1.6?
>> >
>> > Digging a bit deeper in the log files with
>> > log4j.category.org.apache.axis2=DEBUG and
>> > log4j.category.org.apache.ode=DEBUG configured yields interesting
>> results.
>> > When a ServiceClient instance is created, the second parameter is NULL.
>> > When this is the case, Axis2 will create a new anonymous service (
>> >
>> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L137
>> ).
>> > The anonymous service is then added to the AxisConfiguration instance (
>> >
>> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L176
>> > ).
>> >
>> > Internally, this creates a new AxisServiceGroup (
>> >
>> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L315
>> ).
>> > This then adds the new anonymous service to an allEndpoints property (
>> >
>> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L373
>> > ).
>> >
>> > When ODE calls setAxisService, Axis2 clean up the service (
>> >
>> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L612
>> ).
>> > It does not, however, clean up the reference from allEndpoints.
>> >
>> > The filtered catalina.out
>> > <
>> https://gist.githubusercontent.com/StanAngeloff/9f1798e34963012bc190/raw/catalina.out.filtered
>> >
>> > log is attached for reference. Analysing the heap dumps indicates that
>> > allEndpoints is indeed the culprit.
>> >
>> > Axis 1.6 seems to have a fix for this issue:
>> >
>> >
>> https://github.com/apache/axis2-java/blob/1_6/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L679
>> >
>> > Is it possible to use Axis 1.6 in ODE? How does one go about building
>> ODE
>> > and bundling it with Axis 1.6?
>> > Should the ODE 1.3.x branch be updated to use the latest stable Axis2?
>> >
>> > Regards,
>> > Stan
>> >
>> >
>> >
>> >
>> > On 14 July 2014 13:45, Stan Angeloff <st...@angeloff.name> wrote:
>> >
>> >> Hello,
>> >>
>> >> I've been playing with ODE for quite a while now and one of my most
>> >> recent attempts tried to run 5, 000 simultaneous instances of one
>> process.
>> >> Unfortunately after a while ODE uses up all of the allowed memory and
>> barfs
>> >> 'Java heap space' continuously in the log.
>> >>
>> >> My set up is very basic, starting from
>> >> https://github.com/vanto/apache-ode-tomcat-bundle (which has been an
>> >> excellent learning tool). The main .bpel file consists of a continuous
>> >> Repeat Until inside of which there are two sequential invoke activities
>> >> calling external SOAP services.
>> >>
>> >> When the 5, 000 instances are started, the memory footprint is fairly
>> >> stable. After a while, though,
>> org.apache.axis2.engine.AxisConfiguration
>> >> accumulates a lot of org.apache.axis2.description.AxisService instances
>> >> (all of which have their name property set to 'anonService' + hash).
>> >> Those keep growing until eventually the server stops responding due to
>> >> exhausted memory. I have taken heap dumps at various stages and
>> examined
>> >> those. Eclipse was even smart enough to detect the potential memory
>> leak on
>> >> the latest dump just as the server crashed.
>> >>
>> >> One instance of "org.apache.axis2.engine.AxisConfiguration" loaded by
>> >>> "org.apache.catalina.loader.WebappClassLoader @ 0xe07e53e0" occupies
>> >>> 390,800,064 (75.91%) bytes. The memory is accumulated in one instance
>> of
>> >>> "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "<system
>> class
>> >>> loader>".
>> >>
>> >>
>> >> I'll be happy to export the complete project, note however it doesn't
>> >> matter what services are invoked or what the .bpel does so long it's a)
>> >> continuous, e.g., does not stop b) calls an external service.
>> >>
>> >> I tried looking into ODE/Axis2 to try and figure out what might be
>> going
>> >> wrong. My understanding is that service clients are cached and recycled
>> >> in SoapExternalService.java (
>> >>
>> https://github.com/apache/ode/blob/master/axis2/src/main/java/org/apache/ode/axis2/SoapExternalService.java#L265
>> )
>> >> When a new ServiceClient instance is created, it calls the
>> >> configureServiceClient method within Axis2 with axisService = null (
>> >>
>> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L143
>> ).
>> >> That in turn calls createAnonymousService. Later on, ODE calls
>> >> setAxisService which forces Axis2 to dispose of the anonymous service
>> it
>> >> created earlier.
>> >>
>> >> If the above is indeed what is happening, then there should be no
>> memory
>> >> leaks. However, in the memory dumps the number of instances of
>> >> org.apache.axis2.description.AxisService keeps growing. I've tried
>> >> increasing the maximum heap size up to 8GB and it will fail again, it
>> is
>> >> just a matter of time.
>> >>
>> >> Has anyone else experienced this? Is anyone interested in trying to
>> help
>> >> me narrow down the cause of this? Is it possible it's not even a memory
>> >> issue?
>> >>
>> >> Regards,
>> >> Stan
>> >>
>> >
>> >
>>
>
>
Re: Possible memory leak for long-running continuous instances?
Posted by Sathwik B P <sa...@gmail.com>.
Hi Stan,
Thanks for reporting and your patch.
Will look into it when I get some freetime.
For your question about building ODE with axis2 1.6, it's not possible at
the moment as it needs some changes in the Axiom usage within ODE.
Will see if we can bring it on 1.3.7 release, though the timelines are not
fixed yet.
regards,
sathwik
On Mon, Jul 14, 2014 at 10:15 PM, Stan Angeloff <st...@angeloff.name>
wrote:
> For those interested, I ended up patching Axis2 1.5.6 from source with
>
> https://gist.github.com/StanAngeloff/e245a1a349d04efcfad6#file-axis2-1-5-6-cleanup-patch
> The patch has been adapted from two commits on the 1_6 branch. I
> re-compiled the kernel module and swapped the one that comes with the
> binary distribution of ODE. This seems to have resolved the issue for me.
>
> I also found https://issues.apache.org/jira/browse/ODE-585, however it's
> tagged for 2.0. IMHO memory leaks are a pretty major issue and should get
> addressed in whatever is the current stable release?
>
>
> On 14 July 2014 14:31, Stan Angeloff <st...@angeloff.name> wrote:
>
> > *tl;dr;* Is it possible to use Axis 1.6 in ODE? How does one go about
> > building ODE and bundling it with Axis 1.6?
> >
> > Digging a bit deeper in the log files with
> > log4j.category.org.apache.axis2=DEBUG and
> > log4j.category.org.apache.ode=DEBUG configured yields interesting
> results.
> > When a ServiceClient instance is created, the second parameter is NULL.
> > When this is the case, Axis2 will create a new anonymous service (
> >
> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L137
> ).
> > The anonymous service is then added to the AxisConfiguration instance (
> >
> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L176
> > ).
> >
> > Internally, this creates a new AxisServiceGroup (
> >
> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L315
> ).
> > This then adds the new anonymous service to an allEndpoints property (
> >
> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L373
> > ).
> >
> > When ODE calls setAxisService, Axis2 clean up the service (
> >
> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L612
> ).
> > It does not, however, clean up the reference from allEndpoints.
> >
> > The filtered catalina.out
> > <
> https://gist.githubusercontent.com/StanAngeloff/9f1798e34963012bc190/raw/catalina.out.filtered
> >
> > log is attached for reference. Analysing the heap dumps indicates that
> > allEndpoints is indeed the culprit.
> >
> > Axis 1.6 seems to have a fix for this issue:
> >
> >
> https://github.com/apache/axis2-java/blob/1_6/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L679
> >
> > Is it possible to use Axis 1.6 in ODE? How does one go about building ODE
> > and bundling it with Axis 1.6?
> > Should the ODE 1.3.x branch be updated to use the latest stable Axis2?
> >
> > Regards,
> > Stan
> >
> >
> >
> >
> > On 14 July 2014 13:45, Stan Angeloff <st...@angeloff.name> wrote:
> >
> >> Hello,
> >>
> >> I've been playing with ODE for quite a while now and one of my most
> >> recent attempts tried to run 5, 000 simultaneous instances of one
> process.
> >> Unfortunately after a while ODE uses up all of the allowed memory and
> barfs
> >> 'Java heap space' continuously in the log.
> >>
> >> My set up is very basic, starting from
> >> https://github.com/vanto/apache-ode-tomcat-bundle (which has been an
> >> excellent learning tool). The main .bpel file consists of a continuous
> >> Repeat Until inside of which there are two sequential invoke activities
> >> calling external SOAP services.
> >>
> >> When the 5, 000 instances are started, the memory footprint is fairly
> >> stable. After a while, though, org.apache.axis2.engine.AxisConfiguration
> >> accumulates a lot of org.apache.axis2.description.AxisService instances
> >> (all of which have their name property set to 'anonService' + hash).
> >> Those keep growing until eventually the server stops responding due to
> >> exhausted memory. I have taken heap dumps at various stages and examined
> >> those. Eclipse was even smart enough to detect the potential memory
> leak on
> >> the latest dump just as the server crashed.
> >>
> >> One instance of "org.apache.axis2.engine.AxisConfiguration" loaded by
> >>> "org.apache.catalina.loader.WebappClassLoader @ 0xe07e53e0" occupies
> >>> 390,800,064 (75.91%) bytes. The memory is accumulated in one instance
> of
> >>> "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "<system
> class
> >>> loader>".
> >>
> >>
> >> I'll be happy to export the complete project, note however it doesn't
> >> matter what services are invoked or what the .bpel does so long it's a)
> >> continuous, e.g., does not stop b) calls an external service.
> >>
> >> I tried looking into ODE/Axis2 to try and figure out what might be going
> >> wrong. My understanding is that service clients are cached and recycled
> >> in SoapExternalService.java (
> >>
> https://github.com/apache/ode/blob/master/axis2/src/main/java/org/apache/ode/axis2/SoapExternalService.java#L265
> )
> >> When a new ServiceClient instance is created, it calls the
> >> configureServiceClient method within Axis2 with axisService = null (
> >>
> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L143
> ).
> >> That in turn calls createAnonymousService. Later on, ODE calls
> >> setAxisService which forces Axis2 to dispose of the anonymous service it
> >> created earlier.
> >>
> >> If the above is indeed what is happening, then there should be no memory
> >> leaks. However, in the memory dumps the number of instances of
> >> org.apache.axis2.description.AxisService keeps growing. I've tried
> >> increasing the maximum heap size up to 8GB and it will fail again, it is
> >> just a matter of time.
> >>
> >> Has anyone else experienced this? Is anyone interested in trying to help
> >> me narrow down the cause of this? Is it possible it's not even a memory
> >> issue?
> >>
> >> Regards,
> >> Stan
> >>
> >
> >
>
Re: Possible memory leak for long-running continuous instances?
Posted by Stan Angeloff <st...@angeloff.name>.
For those interested, I ended up patching Axis2 1.5.6 from source with
https://gist.github.com/StanAngeloff/e245a1a349d04efcfad6#file-axis2-1-5-6-cleanup-patch
The patch has been adapted from two commits on the 1_6 branch. I
re-compiled the kernel module and swapped the one that comes with the
binary distribution of ODE. This seems to have resolved the issue for me.
I also found https://issues.apache.org/jira/browse/ODE-585, however it's
tagged for 2.0. IMHO memory leaks are a pretty major issue and should get
addressed in whatever is the current stable release?
On 14 July 2014 14:31, Stan Angeloff <st...@angeloff.name> wrote:
> *tl;dr;* Is it possible to use Axis 1.6 in ODE? How does one go about
> building ODE and bundling it with Axis 1.6?
>
> Digging a bit deeper in the log files with
> log4j.category.org.apache.axis2=DEBUG and
> log4j.category.org.apache.ode=DEBUG configured yields interesting results.
> When a ServiceClient instance is created, the second parameter is NULL.
> When this is the case, Axis2 will create a new anonymous service (
> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L137).
> The anonymous service is then added to the AxisConfiguration instance (
> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L176
> ).
>
> Internally, this creates a new AxisServiceGroup (
> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L315).
> This then adds the new anonymous service to an allEndpoints property (
> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L373
> ).
>
> When ODE calls setAxisService, Axis2 clean up the service (
> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L612).
> It does not, however, clean up the reference from allEndpoints.
>
> The filtered catalina.out
> <https://gist.githubusercontent.com/StanAngeloff/9f1798e34963012bc190/raw/catalina.out.filtered>
> log is attached for reference. Analysing the heap dumps indicates that
> allEndpoints is indeed the culprit.
>
> Axis 1.6 seems to have a fix for this issue:
>
> https://github.com/apache/axis2-java/blob/1_6/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L679
>
> Is it possible to use Axis 1.6 in ODE? How does one go about building ODE
> and bundling it with Axis 1.6?
> Should the ODE 1.3.x branch be updated to use the latest stable Axis2?
>
> Regards,
> Stan
>
>
>
>
> On 14 July 2014 13:45, Stan Angeloff <st...@angeloff.name> wrote:
>
>> Hello,
>>
>> I've been playing with ODE for quite a while now and one of my most
>> recent attempts tried to run 5, 000 simultaneous instances of one process.
>> Unfortunately after a while ODE uses up all of the allowed memory and barfs
>> 'Java heap space' continuously in the log.
>>
>> My set up is very basic, starting from
>> https://github.com/vanto/apache-ode-tomcat-bundle (which has been an
>> excellent learning tool). The main .bpel file consists of a continuous
>> Repeat Until inside of which there are two sequential invoke activities
>> calling external SOAP services.
>>
>> When the 5, 000 instances are started, the memory footprint is fairly
>> stable. After a while, though, org.apache.axis2.engine.AxisConfiguration
>> accumulates a lot of org.apache.axis2.description.AxisService instances
>> (all of which have their name property set to 'anonService' + hash).
>> Those keep growing until eventually the server stops responding due to
>> exhausted memory. I have taken heap dumps at various stages and examined
>> those. Eclipse was even smart enough to detect the potential memory leak on
>> the latest dump just as the server crashed.
>>
>> One instance of "org.apache.axis2.engine.AxisConfiguration" loaded by
>>> "org.apache.catalina.loader.WebappClassLoader @ 0xe07e53e0" occupies
>>> 390,800,064 (75.91%) bytes. The memory is accumulated in one instance of
>>> "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "<system class
>>> loader>".
>>
>>
>> I'll be happy to export the complete project, note however it doesn't
>> matter what services are invoked or what the .bpel does so long it's a)
>> continuous, e.g., does not stop b) calls an external service.
>>
>> I tried looking into ODE/Axis2 to try and figure out what might be going
>> wrong. My understanding is that service clients are cached and recycled
>> in SoapExternalService.java (
>> https://github.com/apache/ode/blob/master/axis2/src/main/java/org/apache/ode/axis2/SoapExternalService.java#L265)
>> When a new ServiceClient instance is created, it calls the
>> configureServiceClient method within Axis2 with axisService = null (
>> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L143).
>> That in turn calls createAnonymousService. Later on, ODE calls
>> setAxisService which forces Axis2 to dispose of the anonymous service it
>> created earlier.
>>
>> If the above is indeed what is happening, then there should be no memory
>> leaks. However, in the memory dumps the number of instances of
>> org.apache.axis2.description.AxisService keeps growing. I've tried
>> increasing the maximum heap size up to 8GB and it will fail again, it is
>> just a matter of time.
>>
>> Has anyone else experienced this? Is anyone interested in trying to help
>> me narrow down the cause of this? Is it possible it's not even a memory
>> issue?
>>
>> Regards,
>> Stan
>>
>
>
Re: Possible memory leak for long-running continuous instances?
Posted by Stan Angeloff <st...@angeloff.name>.
*tl;dr;* Is it possible to use Axis 1.6 in ODE? How does one go about
building ODE and bundling it with Axis 1.6?
Digging a bit deeper in the log files with
log4j.category.org.apache.axis2=DEBUG and
log4j.category.org.apache.ode=DEBUG configured yields interesting results.
When a ServiceClient instance is created, the second parameter is NULL.
When this is the case, Axis2 will create a new anonymous service (
https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L137).
The anonymous service is then added to the AxisConfiguration instance (
https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L176
).
Internally, this creates a new AxisServiceGroup (
https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L315).
This then adds the new anonymous service to an allEndpoints property (
https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L373
).
When ODE calls setAxisService, Axis2 clean up the service (
https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L612).
It does not, however, clean up the reference from allEndpoints.
The filtered catalina.out
<https://gist.githubusercontent.com/StanAngeloff/9f1798e34963012bc190/raw/catalina.out.filtered>
log is attached for reference. Analysing the heap dumps indicates that
allEndpoints is indeed the culprit.
Axis 1.6 seems to have a fix for this issue:
https://github.com/apache/axis2-java/blob/1_6/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L679
Is it possible to use Axis 1.6 in ODE? How does one go about building ODE
and bundling it with Axis 1.6?
Should the ODE 1.3.x branch be updated to use the latest stable Axis2?
Regards,
Stan
On 14 July 2014 13:45, Stan Angeloff <st...@angeloff.name> wrote:
> Hello,
>
> I've been playing with ODE for quite a while now and one of my most recent
> attempts tried to run 5, 000 simultaneous instances of one process.
> Unfortunately after a while ODE uses up all of the allowed memory and barfs
> 'Java heap space' continuously in the log.
>
> My set up is very basic, starting from
> https://github.com/vanto/apache-ode-tomcat-bundle (which has been an
> excellent learning tool). The main .bpel file consists of a continuous
> Repeat Until inside of which there are two sequential invoke activities
> calling external SOAP services.
>
> When the 5, 000 instances are started, the memory footprint is fairly
> stable. After a while, though, org.apache.axis2.engine.AxisConfiguration
> accumulates a lot of org.apache.axis2.description.AxisService instances
> (all of which have their name property set to 'anonService' + hash).
> Those keep growing until eventually the server stops responding due to
> exhausted memory. I have taken heap dumps at various stages and examined
> those. Eclipse was even smart enough to detect the potential memory leak on
> the latest dump just as the server crashed.
>
> One instance of "org.apache.axis2.engine.AxisConfiguration" loaded by
>> "org.apache.catalina.loader.WebappClassLoader @ 0xe07e53e0" occupies
>> 390,800,064 (75.91%) bytes. The memory is accumulated in one instance of
>> "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "<system class
>> loader>".
>
>
> I'll be happy to export the complete project, note however it doesn't
> matter what services are invoked or what the .bpel does so long it's a)
> continuous, e.g., does not stop b) calls an external service.
>
> I tried looking into ODE/Axis2 to try and figure out what might be going
> wrong. My understanding is that service clients are cached and recycled
> in SoapExternalService.java (
> https://github.com/apache/ode/blob/master/axis2/src/main/java/org/apache/ode/axis2/SoapExternalService.java#L265)
> When a new ServiceClient instance is created, it calls the
> configureServiceClient method within Axis2 with axisService = null (
> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L143).
> That in turn calls createAnonymousService. Later on, ODE calls
> setAxisService which forces Axis2 to dispose of the anonymous service it
> created earlier.
>
> If the above is indeed what is happening, then there should be no memory
> leaks. However, in the memory dumps the number of instances of
> org.apache.axis2.description.AxisService keeps growing. I've tried
> increasing the maximum heap size up to 8GB and it will fail again, it is
> just a matter of time.
>
> Has anyone else experienced this? Is anyone interested in trying to help
> me narrow down the cause of this? Is it possible it's not even a memory
> issue?
>
> Regards,
> Stan
>