You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@servicemix.apache.org by Felipe Gaucho <fg...@gmail.com> on 2007/03/12 09:04:37 UTC
out of memory - JVM shutdown
I made some proof of concepts with ServiceMixm including a weekend running.
environment:
- 1 test machine (1Gb RAM)
- 1 oracle 8i db in other machine
ServiceMix 3.2 snapshot
Java 1.6.0
Prototype:
- 1 J2SE application that generates 50 records per second into the database
- 1 ServiceMix instance running my prototype code
Design:
- the prototype uses 3 lightweight components:
[input binding] - this component read each new record from DB and generates
a normalized message.
public class GansJdbcDataImporter extends PollingComponentSupport{...}
[service engine] - this component read the message generated by the reader
and create a new normalized message with some content (simulating a business
process)
public class ThresholdEngine extends OutBinding {...}
[output binding] - this component reads the message from the engine and then
writes some log into a file.
public class SmsHttpConnector extends OutBinding{...}
All components writes a timestamp in a monitor table - in order to give me a
glimpsy about the performance of the router and also to give me a chance to
followup the messages.
Test results:
the J2SE application generated 2.5 million records into the database
the JBI container processed 85 thousand messages - and then had problems of
memory.
Evaluation:
it seems the components are not removing the messages from the router after
processing it. Other curious problem is that the stack trace show excess of
threads. I supose there is a pool taking care of the number of threads and
components, so I cannot figure out why the system shutted-down after some
hours.
Possible error causes:
- wrong components code. The type of the components I didn't figured out
very well..
the stack trace:
> ERROR - PollingComponentSupport - Failed to schedule work:
> java.lang.OutO
> fMemoryError: unable to create new native thread
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:597)
> at
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.add
> IfUnderMaximumPoolSize(ThreadPoolExecutor.java:477)
> at
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.exe
> cute(ThreadPoolExecutor.java:876)
> at
> org.apache.servicemix.executors.impl.ExecutorImpl.execute(ExecutorImp
> l.java:42)
> at
> org.apache.servicemix.components.util.PollingComponentSupport$PollSch
> edulerTask.run(PollingComponentSupport.java:163)
> at
> org.apache.servicemix.components.varscheduler.Scheduler$SchedulerTime
> rTask.run(Scheduler.java:108)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
> #
> # An unexpected error has been detected by Java Runtime Environment:
> #
> # java.lang.OutOfMemoryError: requested 1024000 bytes for GrET in
> C:\BUILD_AREA\
> jdk6\hotspot\src\share\vm\utilities\growableArray.cpp. Out of swap space?
> #
> # Internal Error (414C4C4F434154494F4E0E494E4C494E450E4850500017),
> pid=12324, t
> id=12400
> #
> # Java VM: Java HotSpot(TM) Server VM (1.6.0-b105 mixed mode)
> # An error report file with more information is saved as
> hs_err_pid12324.log
> #
> # If you would like to submit a bug report, please visit:
> # http://java.sun.com/webapps/bugreport/crash.jsp
> #
>
--
View this message in context: http://www.nabble.com/out-of-memory---JVM-shutdown-tf3387721s12049.html#a9429944
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: [implentation on an internal system host] out of memory - JVM
shutdown
Posted by Felipe Gaucho <fg...@gmail.com>.
A better look on the output. The components are writing some debug lines in
the console .. and it seems they are consuming the same message more than 1
time - it seems they are not removing the messages of the router after
consume it or other problem.
> Loading Apache ServiceMix from file: servicemix.xml
> INFO - JBIContainer - ServiceMix
> 3.2-incubating-SNAPSHOT JBI
> Container (ServiceMix) is starting
> INFO - JBIContainer - For help or more informations
> please se
> e: http://incubator.apache.org/servicemix/
> INFO - ConnectorServerFactoryBean - JMX connector available at:
> service:jmx
> :rmi:///jndi/rmi://localhost:1099/jmxrmi
> INFO - ComponentMBeanImpl - Initializing component:
> #SubscriptionMa
> nager#
> INFO - DeploymentService - Restoring service assemblies
> INFO - ComponentMBeanImpl - Initializing component:
> fileSender
> INFO - ComponentMBeanImpl - Initializing component:
> filePoller
> INFO - ComponentMBeanImpl - Initializing component:
> smsHttpConnecto
> r
> INFO - JBIContainer - ServiceMix JBI Container
> (ServiceMix) s
> tarted
> SMS sent to 84 about the transaction 83400 on Mo, 12 Mrz 2007 14:15:59
> SMS sent to 4 about the transaction 83402 on Mo, 12 Mrz 2007 14:15:59
> TR 84 processed on Mo, 12 Mrz 2007 14:15:59
> TR 4 processed on Mo, 12 Mrz 2007 14:15:59 #4 ? it was already consumed
> two days ago.
> TR 83400 sent on Mo, 12 Mrz 2007 14:15:59
> TR 83402 sent on Mo, 12 Mrz 2007 14:15:59
> SMS sent to 8 about the transaction 83401 on Mo, 12 Mrz 2007 14:15:59
> SMS sent to 4 about the transaction 83402 on Mo, 12 Mrz 2007 14:15:59
> TR 8 processed on Mo, 12 Mrz 2007 14:15:59
> TR 4 processed on Mo, 12 Mrz 2007 14:15:59
> SMS sent to 4 about the transaction 83402 on Mo, 12 Mrz 2007 14:15:59
> <-- same transaction
> SMS sent to 4 about the transaction 83402 on Mo, 12 Mrz 2007 14:16:00
> <-- three times
> SMS sent to 4 about the transaction 83402 on Mo, 12 Mrz 2007 14:16:00
> <-- ????
> TR 4 processed on Mo, 12 Mrz 2007 14:16:00
> TR 4 processed on Mo, 12 Mrz 2007 14:16:00 <-- here our number 4 again ?
> SMS sent to 8 about the transaction 83401 on Mo, 12 Mrz 2007 14:16:00
> TR 83402 sent on Mo, 12 Mrz 2007 14:16:00
> TR 8 processed on Mo, 12 Mrz 2007 14:16:00
> SMS sent to 4 about the transaction 83403 on Mo, 12 Mrz 2007 14:16:00
>
--
View this message in context: http://www.nabble.com/out-of-memory---JVM-shutdown-tf3387721s12049.html#a9434436
Sent from the ServiceMix - User mailing list archive at Nabble.com.
Re: out of memory - JVM shutdown
Posted by Guillaume Nodet <gn...@gmail.com>.
Threads takes a lot of memory.
The first thing is to make sure your threads are not deadlocked somehow.
I would recommend reducing the number of threads used
(see http://incubator.apache.org/servicemix/thread-pools.html)
and monitoring the process with jconsole. The number of threads
should remain constant. You can also reduce the amount of memory
so that the problem comes faster ...
On 3/12/07, Felipe Gaucho <fg...@gmail.com> wrote:
>
>
> I made some proof of concepts with ServiceMixm including a weekend
> running.
>
> environment:
> - 1 test machine (1Gb RAM)
> - 1 oracle 8i db in other machine
> ServiceMix 3.2 snapshot
> Java 1.6.0
>
> Prototype:
> - 1 J2SE application that generates 50 records per second into the
> database
> - 1 ServiceMix instance running my prototype code
>
> Design:
> - the prototype uses 3 lightweight components:
>
> [input binding] - this component read each new record from DB and
> generates
> a normalized message.
> public class GansJdbcDataImporter extends PollingComponentSupport{...}
>
> [service engine] - this component read the message generated by the reader
> and create a new normalized message with some content (simulating a
> business
> process)
> public class ThresholdEngine extends OutBinding {...}
>
> [output binding] - this component reads the message from the engine and
> then
> writes some log into a file.
> public class SmsHttpConnector extends OutBinding{...}
>
> All components writes a timestamp in a monitor table - in order to give me
> a
> glimpsy about the performance of the router and also to give me a chance
> to
> followup the messages.
>
>
> Test results:
> the J2SE application generated 2.5 million records into the database
> the JBI container processed 85 thousand messages - and then had problems
> of
> memory.
>
> Evaluation:
> it seems the components are not removing the messages from the router
> after
> processing it. Other curious problem is that the stack trace show excess
> of
> threads. I supose there is a pool taking care of the number of threads and
> components, so I cannot figure out why the system shutted-down after some
> hours.
>
> Possible error causes:
> - wrong components code. The type of the components I didn't figured out
> very well..
>
> the stack trace:
>
>
> > ERROR - PollingComponentSupport - Failed to schedule work:
> > java.lang.OutO
> > fMemoryError: unable to create new native thread
> > java.lang.OutOfMemoryError: unable to create new native thread
> > at java.lang.Thread.start0(Native Method)
> > at java.lang.Thread.start(Thread.java:597)
> > at
> > edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.add
> > IfUnderMaximumPoolSize(ThreadPoolExecutor.java:477)
> > at
> > edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.exe
> > cute(ThreadPoolExecutor.java:876)
> > at
> > org.apache.servicemix.executors.impl.ExecutorImpl.execute(ExecutorImp
> > l.java:42)
> > at
> > org.apache.servicemix.components.util.PollingComponentSupport$PollSch
> > edulerTask.run(PollingComponentSupport.java:163)
> > at
> > org.apache.servicemix.components.varscheduler.Scheduler$SchedulerTime
> > rTask.run(Scheduler.java:108)
> > at java.util.TimerThread.mainLoop(Timer.java:512)
> > at java.util.TimerThread.run(Timer.java:462)
> > #
> > # An unexpected error has been detected by Java Runtime Environment:
> > #
> > # java.lang.OutOfMemoryError: requested 1024000 bytes for GrET in
> > C:\BUILD_AREA\
> > jdk6\hotspot\src\share\vm\utilities\growableArray.cpp. Out of swap
> space?
> > #
> > # Internal Error (414C4C4F434154494F4E0E494E4C494E450E4850500017),
> > pid=12324, t
> > id=12400
> > #
> > # Java VM: Java HotSpot(TM) Server VM (1.6.0-b105 mixed mode)
> > # An error report file with more information is saved as
> > hs_err_pid12324.log
> > #
> > # If you would like to submit a bug report, please visit:
> > # http://java.sun.com/webapps/bugreport/crash.jsp
> > #
> >
>
> --
> View this message in context:
> http://www.nabble.com/out-of-memory---JVM-shutdown-tf3387721s12049.html#a9429944
> Sent from the ServiceMix - User mailing list archive at Nabble.com.
>
>
--
Cheers,
Guillaume Nodet
------------------------
Architect, LogicBlaze (http://www.logicblaze.com/)
Blog: http://gnodet.blogspot.com/