You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by sub3 <st...@subwest.com> on 2008/08/11 23:08:05 UTC
Under IO Load
I have a system where 1 is sending to several (a & b) and occasionally b
sends to c (all queues). Very simple. Persistence is set to false. Queue
sizes are capped at 50mb.
If I put the system under a heavy i/o load (several programs reading/writing
lots of files), ActiveMQ will stop.
Am I hitting some system resource issue here, and if so, what is it? Also,
is there a way to get ActiveMQ to recover?
I've attached my activemq.xml file, and a sample program to create i/o load
(sometime I run several of these). I am running on Windows Server 2003.
Thanks.
http://www.nabble.com/file/p18933907/activemq.xml activemq.xml
http://www.nabble.com/file/p18933907/Test.java Test.java
--
View this message in context: http://www.nabble.com/Under-IO-Load-tp18933907p18933907.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Under IO Load
Posted by sub3 <st...@subwest.com>.
After pouring through more debug, this one seems to come up occasionally:
2008-08-13 12:26:50,144 [InactivityMonitor Async Task:
java.util.concurrent.ThreadPoolExecutor$Worker@1e6ba8ee] DEBUG
org.apache.activemq.ActiveMQConnection - Async exception with no exception
listener: org.apache.activemq.transport.InactivityIOException: Channel was
inactive for too long: localhost/127.0.0.1:61616
Could this be causing an issue, possibly timing something out?
Thanks.
--
View this message in context: http://www.nabble.com/Under-IO-Load-tp18933907p18966891.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Under IO Load
Posted by sub3 <st...@subwest.com>.
Software installed on C:
Heavy I/O is happening on E:
C: is 10k 74gb drive
E: is 2 15k 750gb drives, raid 0
It is an HP server, w/2 xeon cpus. I don't remember the speed, but it is
about a year old.
It is running Windows 2003 server w/ over 8 gigs of ram. We never have a
memory problem.
Hope that helps. Thanks.
elihusmails wrote:
>
> what is your hardware configuration?
>
>
> On Mon, Aug 11, 2008 at 8:23 PM, sub3 <st...@subwest.com> wrote:
>
>>
>> Sorry, that last sentence sounds wrong, let me restate:
>> When I run several of those Test.java programs I attached, ActiveMQ
>> stops.
>> W/o these running ActiveMQ is fine.
>> So I don't know if it is a system resource issue, if so, what?
>> Or it is a condition that only arises when the system is bogged down with
>> i/o.
>>
>> Any insight would be appreciated. Thanks.
>>
>>
>> sub3 wrote:
>> >
>> > That is correct. The processes that are performing the i/o on the disk
>> > are separate jvms. It is not the same as the components using
>> ActiveMQ.
>> > They high i/o usage are somehow causing ActiveMQ to stop. I've run
>> many
>> > tests, with and without them running, ActiveMQ halts w/o any
>> > warnings/errors & it never recovers.
>> >
>> >
>> > elihusmails wrote:
>> >>
>> >> So if I understand this right, you are sending messages to ActiveMQ
>> while
>> >> performing I/O, and ActiveMQ is not performing any I/O (other than
>> >> caching
>> >> of data) since you are not persisting any data.
>> >>
>> >> Is this true?
>> >>
>> >>
>> >>
>> >> On Mon, Aug 11, 2008 at 5:08 PM, sub3 <st...@subwest.com> wrote:
>> >>
>> >>>
>> >>> I have a system where 1 is sending to several (a & b) and
>> occasionally
>> b
>> >>> sends to c (all queues). Very simple. Persistence is set to false.
>> >>> Queue
>> >>> sizes are capped at 50mb.
>> >>>
>> >>> If I put the system under a heavy i/o load (several programs
>> >>> reading/writing
>> >>> lots of files), ActiveMQ will stop.
>> >>>
>> >>> Am I hitting some system resource issue here, and if so, what is it?
>> >>> Also,
>> >>> is there a way to get ActiveMQ to recover?
>> >>>
>> >>> I've attached my activemq.xml file, and a sample program to create
>> i/o
>> >>> load
>> >>> (sometime I run several of these). I am running on Windows Server
>> 2003.
>> >>>
>> >>> Thanks.
>> >>> http://www.nabble.com/file/p18933907/activemq.xml activemq.xml
>> >>> http://www.nabble.com/file/p18933907/Test.java Test.java
>> >>> --
>> >>> View this message in context:
>> >>> http://www.nabble.com/Under-IO-Load-tp18933907p18933907.html
>> >>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>> >>>
>> >>>
>> >>
>> >>
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Under-IO-Load-tp18933907p18936148.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Under-IO-Load-tp18933907p18943417.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Under IO Load
Posted by Mark Webb <el...@gmail.com>.
what is your hardware configuration?
On Mon, Aug 11, 2008 at 8:23 PM, sub3 <st...@subwest.com> wrote:
>
> Sorry, that last sentence sounds wrong, let me restate:
> When I run several of those Test.java programs I attached, ActiveMQ stops.
> W/o these running ActiveMQ is fine.
> So I don't know if it is a system resource issue, if so, what?
> Or it is a condition that only arises when the system is bogged down with
> i/o.
>
> Any insight would be appreciated. Thanks.
>
>
> sub3 wrote:
> >
> > That is correct. The processes that are performing the i/o on the disk
> > are separate jvms. It is not the same as the components using ActiveMQ.
> > They high i/o usage are somehow causing ActiveMQ to stop. I've run many
> > tests, with and without them running, ActiveMQ halts w/o any
> > warnings/errors & it never recovers.
> >
> >
> > elihusmails wrote:
> >>
> >> So if I understand this right, you are sending messages to ActiveMQ
> while
> >> performing I/O, and ActiveMQ is not performing any I/O (other than
> >> caching
> >> of data) since you are not persisting any data.
> >>
> >> Is this true?
> >>
> >>
> >>
> >> On Mon, Aug 11, 2008 at 5:08 PM, sub3 <st...@subwest.com> wrote:
> >>
> >>>
> >>> I have a system where 1 is sending to several (a & b) and occasionally
> b
> >>> sends to c (all queues). Very simple. Persistence is set to false.
> >>> Queue
> >>> sizes are capped at 50mb.
> >>>
> >>> If I put the system under a heavy i/o load (several programs
> >>> reading/writing
> >>> lots of files), ActiveMQ will stop.
> >>>
> >>> Am I hitting some system resource issue here, and if so, what is it?
> >>> Also,
> >>> is there a way to get ActiveMQ to recover?
> >>>
> >>> I've attached my activemq.xml file, and a sample program to create i/o
> >>> load
> >>> (sometime I run several of these). I am running on Windows Server
> 2003.
> >>>
> >>> Thanks.
> >>> http://www.nabble.com/file/p18933907/activemq.xml activemq.xml
> >>> http://www.nabble.com/file/p18933907/Test.java Test.java
> >>> --
> >>> View this message in context:
> >>> http://www.nabble.com/Under-IO-Load-tp18933907p18933907.html
> >>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>>
> >>>
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Under-IO-Load-tp18933907p18936148.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
Re: Under IO Load
Posted by sub3 <st...@subwest.com>.
Sorry, that last sentence sounds wrong, let me restate:
When I run several of those Test.java programs I attached, ActiveMQ stops.
W/o these running ActiveMQ is fine.
So I don't know if it is a system resource issue, if so, what?
Or it is a condition that only arises when the system is bogged down with
i/o.
Any insight would be appreciated. Thanks.
sub3 wrote:
>
> That is correct. The processes that are performing the i/o on the disk
> are separate jvms. It is not the same as the components using ActiveMQ.
> They high i/o usage are somehow causing ActiveMQ to stop. I've run many
> tests, with and without them running, ActiveMQ halts w/o any
> warnings/errors & it never recovers.
>
>
> elihusmails wrote:
>>
>> So if I understand this right, you are sending messages to ActiveMQ while
>> performing I/O, and ActiveMQ is not performing any I/O (other than
>> caching
>> of data) since you are not persisting any data.
>>
>> Is this true?
>>
>>
>>
>> On Mon, Aug 11, 2008 at 5:08 PM, sub3 <st...@subwest.com> wrote:
>>
>>>
>>> I have a system where 1 is sending to several (a & b) and occasionally b
>>> sends to c (all queues). Very simple. Persistence is set to false.
>>> Queue
>>> sizes are capped at 50mb.
>>>
>>> If I put the system under a heavy i/o load (several programs
>>> reading/writing
>>> lots of files), ActiveMQ will stop.
>>>
>>> Am I hitting some system resource issue here, and if so, what is it?
>>> Also,
>>> is there a way to get ActiveMQ to recover?
>>>
>>> I've attached my activemq.xml file, and a sample program to create i/o
>>> load
>>> (sometime I run several of these). I am running on Windows Server 2003.
>>>
>>> Thanks.
>>> http://www.nabble.com/file/p18933907/activemq.xml activemq.xml
>>> http://www.nabble.com/file/p18933907/Test.java Test.java
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Under-IO-Load-tp18933907p18933907.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Under-IO-Load-tp18933907p18936148.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Under IO Load
Posted by sub3 <st...@subwest.com>.
That is correct. The processes that are performing the i/o on the disk are
separate jvms. It is not the same as the components using ActiveMQ. They
high i/o usage are somehow causing ActiveMQ to stop. I've run many tests,
with and without them running, ActiveMQ halts w/o any warnings/errors & it
never recovers.
elihusmails wrote:
>
> So if I understand this right, you are sending messages to ActiveMQ while
> performing I/O, and ActiveMQ is not performing any I/O (other than caching
> of data) since you are not persisting any data.
>
> Is this true?
>
>
>
> On Mon, Aug 11, 2008 at 5:08 PM, sub3 <st...@subwest.com> wrote:
>
>>
>> I have a system where 1 is sending to several (a & b) and occasionally b
>> sends to c (all queues). Very simple. Persistence is set to false.
>> Queue
>> sizes are capped at 50mb.
>>
>> If I put the system under a heavy i/o load (several programs
>> reading/writing
>> lots of files), ActiveMQ will stop.
>>
>> Am I hitting some system resource issue here, and if so, what is it?
>> Also,
>> is there a way to get ActiveMQ to recover?
>>
>> I've attached my activemq.xml file, and a sample program to create i/o
>> load
>> (sometime I run several of these). I am running on Windows Server 2003.
>>
>> Thanks.
>> http://www.nabble.com/file/p18933907/activemq.xml activemq.xml
>> http://www.nabble.com/file/p18933907/Test.java Test.java
>> --
>> View this message in context:
>> http://www.nabble.com/Under-IO-Load-tp18933907p18933907.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Under-IO-Load-tp18933907p18936118.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Under IO Load
Posted by Mark Webb <el...@gmail.com>.
So if I understand this right, you are sending messages to ActiveMQ while
performing I/O, and ActiveMQ is not performing any I/O (other than caching
of data) since you are not persisting any data.
Is this true?
On Mon, Aug 11, 2008 at 5:08 PM, sub3 <st...@subwest.com> wrote:
>
> I have a system where 1 is sending to several (a & b) and occasionally b
> sends to c (all queues). Very simple. Persistence is set to false. Queue
> sizes are capped at 50mb.
>
> If I put the system under a heavy i/o load (several programs
> reading/writing
> lots of files), ActiveMQ will stop.
>
> Am I hitting some system resource issue here, and if so, what is it? Also,
> is there a way to get ActiveMQ to recover?
>
> I've attached my activemq.xml file, and a sample program to create i/o load
> (sometime I run several of these). I am running on Windows Server 2003.
>
> Thanks.
> http://www.nabble.com/file/p18933907/activemq.xml activemq.xml
> http://www.nabble.com/file/p18933907/Test.java Test.java
> --
> View this message in context:
> http://www.nabble.com/Under-IO-Load-tp18933907p18933907.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
Re: Under IO Load
Posted by sub3 <st...@subwest.com>.
bsnyder wrote:
>
> On Mon, Aug 11, 2008 at 3:08 PM, sub3 <st...@subwest.com> wrote:
>>
>> I have a system where 1 is sending to several (a & b) and occasionally b
>> sends to c (all queues). Very simple. Persistence is set to false.
>> Queue
>> sizes are capped at 50mb.
>>
>> If I put the system under a heavy i/o load (several programs
>> reading/writing
>> lots of files), ActiveMQ will stop.
>>
>> Am I hitting some system resource issue here, and if so, what is it?
>> Also,
>> is there a way to get ActiveMQ to recover?
>>
>> I've attached my activemq.xml file, and a sample program to create i/o
>> load
>> (sometime I run several of these). I am running on Windows Server 2003.
>
> What happens with ActiveMQ under the same messaging load if you do not
> run the Test.java class in other JVMs?
>
> Bruce
> --
> perl -e 'print
> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> Apache ActiveMQ - http://activemq.org/
> Apache Camel - http://activemq.org/camel/
> Apache ServiceMix - http://servicemix.org/
>
> Blog: http://bruceblog.org/
>
>
If I don't run Test.java, the system remains up & stable. I've run it for
over a week as fast as possible w/o any problems.
--
View this message in context: http://www.nabble.com/Under-IO-Load-tp18933907p18943362.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Under IO Load
Posted by Bruce Snyder <br...@gmail.com>.
On Mon, Aug 11, 2008 at 3:08 PM, sub3 <st...@subwest.com> wrote:
>
> I have a system where 1 is sending to several (a & b) and occasionally b
> sends to c (all queues). Very simple. Persistence is set to false. Queue
> sizes are capped at 50mb.
>
> If I put the system under a heavy i/o load (several programs reading/writing
> lots of files), ActiveMQ will stop.
>
> Am I hitting some system resource issue here, and if so, what is it? Also,
> is there a way to get ActiveMQ to recover?
>
> I've attached my activemq.xml file, and a sample program to create i/o load
> (sometime I run several of these). I am running on Windows Server 2003.
What happens with ActiveMQ under the same messaging load if you do not
run the Test.java class in other JVMs?
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
Apache ActiveMQ - http://activemq.org/
Apache Camel - http://activemq.org/camel/
Apache ServiceMix - http://servicemix.org/
Blog: http://bruceblog.org/