You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@servicemix.apache.org by "BURN, James" <Ja...@oup.com> on 2015/06/03 18:11:31 UTC

Controlling multiple ActiveMQ brokers in ServiceMix

I'm trying to pin down why one of our development boxes keeps falling over whilst under heavy processing load.

This is running ServiceMix 5.1.1. We have around 60 different Camel routes running.

I "think" that this ServiceMix has many ActiveMQ brokers running. I see these sort of errors in the logs:

2015-06-03 16:51:04,315 | ERROR | dTransformation] | faultJmsMessageListenerContainer | 118 - org.springframework.jms - 3.2.4.RELEASE | Could not refresh JMS Connection for destination 'STAR.US.SecondTransformation' - retrying in 5000 ms. Cause: Broker: amq-broker - Client: ID:TESTGBOXFAPP22-60260-1433344527710-2:8 already connected from vm://amq-broker#14

>From what I can gather, ActiveMQ brokers are used to distribute the message processing over a network. However, we are running all our processing on a single VM so this shouldn't be necessary.

However, from the ServiceMix console and ActiveMQ web console, I can only see the name of one broker (the one defined in etc/org.apache.activemq.server-default.cfg)

What is causing these other brokers to be created, and how can I manage them/reduce their number in ServiceMix?

Thanks

James

Oxford University Press (UK) Disclaimer

This message is confidential. You should not copy it or disclose its contents to anyone. You may use and apply the information for the intended purpose only. OUP does not accept legal responsibility for the contents of this message. Any views or opinions presented are those of the author only and not of OUP. If this email has come to you in error, please delete it, along with any attachments. Please note that OUP may intercept incoming and outgoing email communications.

Re: Controlling multiple ActiveMQ brokers in ServiceMix

Posted by Sobkowiak Krzysztof <kr...@gmail.com>.

Hi

ServiceMix uses own connection factory pool for the "activemq" Camel endpoint defined here
https://github.com/apache/servicemix/blob/master/activemq/activemq-service/src/main/resources/OSGI-INF/blueprint/activemq-service.xml.  The maxConnections parameter is set
to 8 so I'm not sure why you have 20 open connections. If you have still the problem you can try to define your own connection factory (install the jms feature and use the
jms:create command -- http://karaf.apache.org/manual/latest/users-guide/jms.html), reference the connection factory in your blueprint file, define your own "activemq" Camel
component (with other name, e.g. "amq") and use it in your route.

Regards
Krzysztof

On 04.06.2015 16:36, BURN, James wrote:
> Hi
>
> I managed to stop these errors by clearing out the activemq.DLQ which had got tied up when I restarted SM I think . Once I cleared this out and restarted ServiceMix things started working again.
>
> I'm still left wondering about the number of open ActiveMQ connections and whether our running ActiveMQ out of the box should be tweaked.
>
> I can from my Jconsole see we have around 20 vm://amq-broker# transport threads running at present. Most of these are parked and, I think, not taking much resource. However, I'm concerned if this is a good thing to have?
>
> As we're not using PooledConnectionFactory - what happens through ServiceMix implementation of ActiveMQ by default? 
>
> Should we be referring to the amq-broker by its specific name in each Camel route?
>
> Any guidance welcome.
>
> Cheers
>
> James
>
>
>
>
>
>
>
>
> -----Original Message-----
> From: BURN, James [mailto:James.Burn@oup.com] 
> Sent: 03 June 2015 23:18
> To: users@servicemix.apache.org
> Subject: RE: Controlling multiple ActiveMQ brokers in ServiceMix
>
> Hi
>
> Thanks for helping with this. So the errors are showing connections (eg vm://amq-broker#14) to the (solo) broker?
>
> I'll have to look through the (blueprint) route code in the office tomorrow. From what I recall I don't think we configure any sort of ConnectionFactory - just pushing the messages from topic to topic with something like:
>
>  <to uri="activemq:topic:STAR.US.SecondTransformation"/>
>
> I see from http://camel.apache.org/activemq.html that it's recommended to use PooledConnectionFactory but what happens if we don't set this up - the issues below I guess?! 
>
> I'll do some more reading tonight.
>
> Cheers
>
> James
>
> ________________________________________
> From: Krzysztof Sobkowiak [krzys.sobkowiak@gmail.com]
> Sent: 03 June 2015 19:32
> To: users@servicemix.apache.org
> Subject: Re: Controlling multiple ActiveMQ brokers in ServiceMix
>
> Hi
>
> ServiceMix starts only one broker. I think your error shows a problem while creating the connections to the broker in your routes, exactly for this destination STAR.US.SecondTransformation. Probably when something fails your connections will be not correctly closed. Do you use PooledConnectionFactory?
>
> Regards
> Krzysztof
>
> On 03.06.2015 18:11, BURN, James wrote:
>> Hi
>>
>> I'm trying to pin down why one of our development boxes keeps falling over whilst under heavy processing load.
>>
>> This is running ServiceMix 5.1.1. We have around 60 different Camel routes running.
>>
>> I "think" that this ServiceMix has many ActiveMQ brokers running. I see these sort of errors in the logs:
>>
>> 2015-06-03 16:51:04,315 | ERROR | dTransformation] | 
>> faultJmsMessageListenerContainer | 118 - org.springframework.jms - 
>> 3.2.4.RELEASE | Could not refresh JMS Connection for destination 
>> 'STAR.US.SecondTransformation' - retrying in 5000 ms. Cause: Broker: 
>> amq-broker - Client: ID:TESTGBOXFAPP22-60260-1433344527710-2:8 already 
>> connected from vm://amq-broker#14
>>
>> 2015-06-03 16:51:04,315 | WARN  | /amq-broker#15-1 | TransportConnection              | 97 - org.apache.activemq.activemq-osgi - 5.10.0 | Failed to add Connection ID:TESTGBOXFAPP22-60260-1433344527710-3:8
>>
>> From what I can gather, ActiveMQ brokers are used to distribute the message processing over a network. However, we are running all our processing on a single VM so this shouldn't be necessary.
>>
>> However, from the ServiceMix console and ActiveMQ web console, I can 
>> only see the name of one broker (the one defined in 
>> etc/org.apache.activemq.server-default.cfg)
>>
>> What is causing these other brokers to be created, and how can I manage them/reduce their number in ServiceMix?
>>
>> Thanks
>>
>> James
>>
>>
>>
>> Oxford University Press (UK) Disclaimer
>>
>> This message is confidential. You should not copy it or disclose its contents to anyone. You may use and apply the information for the intended purpose only. OUP does not accept legal responsibility for the contents of this message. Any views or opinions presented are those of the author only and not of OUP. If this email has come to you in error, please delete it, along with any attachments. Please note that OUP may intercept incoming and outgoing email communications.
>>
> --
> Krzysztof Sobkowiak
>
> JEE & OSS Architect
> Apache Software Foundation Member
> Apache ServiceMix <http://servicemix.apache.org/> Committer & PMC chair Senior Solution Architect @ Capgemini SSC <http://www.pl.capgemini-sdm.com/en/>
> Oxford University Press (UK) Disclaimer
>
> This message is confidential. You should not copy it or disclose its contents to anyone. You may use and apply the information for the intended purpose only. OUP does not accept legal responsibility for the contents of this message. Any views or opinions presented are those of the author only and not of OUP. If this email has come to you in error, please delete it, along with any attachments. Please note that OUP may intercept incoming and outgoing email communications.
> Oxford University Press (UK) Disclaimer
>
> This message is confidential. You should not copy it or disclose its contents to anyone. You may use and apply the information for the intended purpose only. OUP does not accept legal responsibility for the contents of this message. Any views or opinions presented are those of the author only and not of OUP. If this email has come to you in error, please delete it, along with any attachments. Please note that OUP may intercept incoming and outgoing email communications.

-- 
Krzysztof Sobkowiak

JEE & OSS Architect
Apache Software Foundation Member
Apache ServiceMix <http://servicemix.apache.org/> Committer & PMC
Senior Solution Architect @ Capgemini SSC <http://www.pl.capgemini-sdm.com/en/>

RE: Controlling multiple ActiveMQ brokers in ServiceMix

Posted by "BURN, James" <Ja...@oup.com>.

Hi

I managed to stop these errors by clearing out the activemq.DLQ which had got tied up when I restarted SM I think . Once I cleared this out and restarted ServiceMix things started working again.

I'm still left wondering about the number of open ActiveMQ connections and whether our running ActiveMQ out of the box should be tweaked.

I can from my Jconsole see we have around 20 vm://amq-broker# transport threads running at present. Most of these are parked and, I think, not taking much resource. However, I'm concerned if this is a good thing to have?

As we're not using PooledConnectionFactory - what happens through ServiceMix implementation of ActiveMQ by default? 

Should we be referring to the amq-broker by its specific name in each Camel route?

Any guidance welcome.

Cheers

James

-----Original Message-----
From: BURN, James [mailto:James.Burn@oup.com] 
Sent: 03 June 2015 23:18
To: users@servicemix.apache.org
Subject: RE: Controlling multiple ActiveMQ brokers in ServiceMix

Hi

Thanks for helping with this. So the errors are showing connections (eg vm://amq-broker#14) to the (solo) broker?

I'll have to look through the (blueprint) route code in the office tomorrow. From what I recall I don't think we configure any sort of ConnectionFactory - just pushing the messages from topic to topic with something like:

 <to uri="activemq:topic:STAR.US.SecondTransformation"/>

I see from http://camel.apache.org/activemq.html that it's recommended to use PooledConnectionFactory but what happens if we don't set this up - the issues below I guess?! 

I'll do some more reading tonight.

Cheers

James

________________________________________
From: Krzysztof Sobkowiak [krzys.sobkowiak@gmail.com]
Sent: 03 June 2015 19:32
To: users@servicemix.apache.org
Subject: Re: Controlling multiple ActiveMQ brokers in ServiceMix

Hi

ServiceMix starts only one broker. I think your error shows a problem while creating the connections to the broker in your routes, exactly for this destination STAR.US.SecondTransformation. Probably when something fails your connections will be not correctly closed. Do you use PooledConnectionFactory?

Regards
Krzysztof

On 03.06.2015 18:11, BURN, James wrote:
> Hi
>
> I'm trying to pin down why one of our development boxes keeps falling over whilst under heavy processing load.
>
> This is running ServiceMix 5.1.1. We have around 60 different Camel routes running.
>
> I "think" that this ServiceMix has many ActiveMQ brokers running. I see these sort of errors in the logs:
>
> 2015-06-03 16:51:04,315 | ERROR | dTransformation] | 
> faultJmsMessageListenerContainer | 118 - org.springframework.jms - 
> 3.2.4.RELEASE | Could not refresh JMS Connection for destination 
> 'STAR.US.SecondTransformation' - retrying in 5000 ms. Cause: Broker: 
> amq-broker - Client: ID:TESTGBOXFAPP22-60260-1433344527710-2:8 already 
> connected from vm://amq-broker#14
>
> 2015-06-03 16:51:04,315 | WARN  | /amq-broker#15-1 | TransportConnection              | 97 - org.apache.activemq.activemq-osgi - 5.10.0 | Failed to add Connection ID:TESTGBOXFAPP22-60260-1433344527710-3:8
>
> From what I can gather, ActiveMQ brokers are used to distribute the message processing over a network. However, we are running all our processing on a single VM so this shouldn't be necessary.
>
> However, from the ServiceMix console and ActiveMQ web console, I can 
> only see the name of one broker (the one defined in 
> etc/org.apache.activemq.server-default.cfg)
>
> What is causing these other brokers to be created, and how can I manage them/reduce their number in ServiceMix?
>
> Thanks
>
> James
>
>
>
> Oxford University Press (UK) Disclaimer
>
> This message is confidential. You should not copy it or disclose its contents to anyone. You may use and apply the information for the intended purpose only. OUP does not accept legal responsibility for the contents of this message. Any views or opinions presented are those of the author only and not of OUP. If this email has come to you in error, please delete it, along with any attachments. Please note that OUP may intercept incoming and outgoing email communications.
>

--
Krzysztof Sobkowiak

JEE & OSS Architect
Apache Software Foundation Member
Apache ServiceMix <http://servicemix.apache.org/> Committer & PMC chair Senior Solution Architect @ Capgemini SSC <http://www.pl.capgemini-sdm.com/en/>
Oxford University Press (UK) Disclaimer

This message is confidential. You should not copy it or disclose its contents to anyone. You may use and apply the information for the intended purpose only. OUP does not accept legal responsibility for the contents of this message. Any views or opinions presented are those of the author only and not of OUP. If this email has come to you in error, please delete it, along with any attachments. Please note that OUP may intercept incoming and outgoing email communications.
Oxford University Press (UK) Disclaimer

This message is confidential. You should not copy it or disclose its contents to anyone. You may use and apply the information for the intended purpose only. OUP does not accept legal responsibility for the contents of this message. Any views or opinions presented are those of the author only and not of OUP. If this email has come to you in error, please delete it, along with any attachments. Please note that OUP may intercept incoming and outgoing email communications.

RE: Controlling multiple ActiveMQ brokers in ServiceMix

Posted by "BURN, James" <Ja...@oup.com>.

Hi

Thanks for helping with this. So the errors are showing connections (eg vm://amq-broker#14) to the (solo) broker?

I'll have to look through the (blueprint) route code in the office tomorrow. From what I recall I don't think we configure any sort of ConnectionFactory - just pushing the messages from topic to topic with something like:

 <to uri="activemq:topic:STAR.US.SecondTransformation"/>

I see from http://camel.apache.org/activemq.html that it's recommended to use PooledConnectionFactory but what happens if we don't set this up - the issues below I guess?! 

I'll do some more reading tonight.

Cheers

James

________________________________________
From: Krzysztof Sobkowiak [krzys.sobkowiak@gmail.com]
Sent: 03 June 2015 19:32
To: users@servicemix.apache.org
Subject: Re: Controlling multiple ActiveMQ brokers in ServiceMix

Hi

ServiceMix starts only one broker. I think your error shows a problem while creating the connections to the broker in
your routes, exactly for this destination STAR.US.SecondTransformation. Probably when something fails your connections
will be not correctly closed. Do you use PooledConnectionFactory?

Regards
Krzysztof

On 03.06.2015 18:11, BURN, James wrote:
> Hi
>
> I'm trying to pin down why one of our development boxes keeps falling over whilst under heavy processing load.
>
> This is running ServiceMix 5.1.1. We have around 60 different Camel routes running.
>
> I "think" that this ServiceMix has many ActiveMQ brokers running. I see these sort of errors in the logs:
>
> 2015-06-03 16:51:04,315 | ERROR | dTransformation] | faultJmsMessageListenerContainer | 118 - org.springframework.jms - 3.2.4.RELEASE | Could not refresh JMS Connection for destination 'STAR.US.SecondTransformation' - retrying in 5000 ms. Cause: Broker: amq-broker - Client: ID:TESTGBOXFAPP22-60260-1433344527710-2:8 already connected from vm://amq-broker#14
>
> 2015-06-03 16:51:04,315 | WARN  | /amq-broker#15-1 | TransportConnection              | 97 - org.apache.activemq.activemq-osgi - 5.10.0 | Failed to add Connection ID:TESTGBOXFAPP22-60260-1433344527710-3:8
>
> From what I can gather, ActiveMQ brokers are used to distribute the message processing over a network. However, we are running all our processing on a single VM so this shouldn't be necessary.
>
> However, from the ServiceMix console and ActiveMQ web console, I can only see the name of one broker (the one defined in etc/org.apache.activemq.server-default.cfg)
>
> What is causing these other brokers to be created, and how can I manage them/reduce their number in ServiceMix?
>
> Thanks
>
> James
>
>
>
> Oxford University Press (UK) Disclaimer
>
> This message is confidential. You should not copy it or disclose its contents to anyone. You may use and apply the information for the intended purpose only. OUP does not accept legal responsibility for the contents of this message. Any views or opinions presented are those of the author only and not of OUP. If this email has come to you in error, please delete it, along with any attachments. Please note that OUP may intercept incoming and outgoing email communications.
>

--
Krzysztof Sobkowiak

JEE & OSS Architect
Apache Software Foundation Member
Apache ServiceMix <http://servicemix.apache.org/> Committer & PMC chair
Senior Solution Architect @ Capgemini SSC <http://www.pl.capgemini-sdm.com/en/>
Oxford University Press (UK) Disclaimer

This message is confidential. You should not copy it or disclose its contents to anyone. You may use and apply the information for the intended purpose only. OUP does not accept legal responsibility for the contents of this message. Any views or opinions presented are those of the author only and not of OUP. If this email has come to you in error, please delete it, along with any attachments. Please note that OUP may intercept incoming and outgoing email communications.

Re: Controlling multiple ActiveMQ brokers in ServiceMix

Posted by Krzysztof Sobkowiak <kr...@gmail.com>.

Hi

ServiceMix starts only one broker. I think your error shows a problem while creating the connections to the broker in
your routes, exactly for this destination STAR.US.SecondTransformation. Probably when something fails your connections
will be not correctly closed. Do you use PooledConnectionFactory?

Regards
Krzysztof

On 03.06.2015 18:11, BURN, James wrote:
> Hi
>
> I'm trying to pin down why one of our development boxes keeps falling over whilst under heavy processing load.
>
> This is running ServiceMix 5.1.1. We have around 60 different Camel routes running.
>
> I "think" that this ServiceMix has many ActiveMQ brokers running. I see these sort of errors in the logs:
>
> 2015-06-03 16:51:04,315 | ERROR | dTransformation] | faultJmsMessageListenerContainer | 118 - org.springframework.jms - 3.2.4.RELEASE | Could not refresh JMS Connection for destination 'STAR.US.SecondTransformation' - retrying in 5000 ms. Cause: Broker: amq-broker - Client: ID:TESTGBOXFAPP22-60260-1433344527710-2:8 already connected from vm://amq-broker#14
>
> 2015-06-03 16:51:04,315 | WARN  | /amq-broker#15-1 | TransportConnection              | 97 - org.apache.activemq.activemq-osgi - 5.10.0 | Failed to add Connection ID:TESTGBOXFAPP22-60260-1433344527710-3:8
>
> From what I can gather, ActiveMQ brokers are used to distribute the message processing over a network. However, we are running all our processing on a single VM so this shouldn't be necessary.
>
> However, from the ServiceMix console and ActiveMQ web console, I can only see the name of one broker (the one defined in etc/org.apache.activemq.server-default.cfg)
>
> What is causing these other brokers to be created, and how can I manage them/reduce their number in ServiceMix?
>
> Thanks
>
> James
>
>
>
> Oxford University Press (UK) Disclaimer
>
> This message is confidential. You should not copy it or disclose its contents to anyone. You may use and apply the information for the intended purpose only. OUP does not accept legal responsibility for the contents of this message. Any views or opinions presented are those of the author only and not of OUP. If this email has come to you in error, please delete it, along with any attachments. Please note that OUP may intercept incoming and outgoing email communications.
>

-- 
Krzysztof Sobkowiak

JEE & OSS Architect
Apache Software Foundation Member
Apache ServiceMix <http://servicemix.apache.org/> Committer & PMC chair
Senior Solution Architect @ Capgemini SSC <http://www.pl.capgemini-sdm.com/en/>