You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by pollotek <cl...@gmail.com> on 2012/08/11 04:20:59 UTC
Store and Forward with embedded brokers + stand alone broker
I have this web application that currently connects to a stand alone broker
to place messages and consumer only nodes pick up and process the messages.
I want to move this to have embedded brokers on each web app and each
consumer and those embedded brokers should forward their messages to a stand
alone broker that will sit in the middle just like it does today.
producer app server -> embedded broker -> stand alone broker -> embedded
broker -> consumer app server
This will allow messages to be produced and queued up locally without
blocking and have the messages forwarded asynchronously. As I understand the
only thing to do here is to declare the embedded brokers on both app servers
and point their network connectors to the stand alone broker. Messages are
supposed to be stored in the embedded broker and if there are subscribers to
the queue on the stand alone broker side messages will be forwarded.
So I configured my embedded brokers with these parameters (this is the same
configuration on both, consumers and producers) :
<amq:transportConnectors>
<amq:transportConnector uri="vm://localhost?async=true" />
</amq:transportConnectors>
<amq:networkConnectors>
<amq:networkConnector
name="connection_to_nob-${activemq.brokerid}"
uri="static:failover:(tcp://broker01:61616,tcp://broker02:61616)?randomize=false&maxReconnectAttempts=0"/>
</amq:networkConnectors>
The stand alone broker of course is configured to receive connections on
61616. I start up my application and make sure with JMX all consumers and
producers are connected to the same broker (Master-slave shared file system)
and everything looks ok so far. I create some messages for queues I have
consumers for declared in the consumer side only.
I point my producer and consumer app servers to connect to vm://localhost. I
never see a single message leave the embedded broker on the producers side.
I can see with JMX the stand alone broker doesn't create the queues where my
process is placing new messages.
As I understand from what I read I don't need to specifically create the
queues to get messages forwarded to in the stand alone broker, they're
supposed to be automatically created. I don't have anything special on my
stand alone broker other than this destination Policy which I also have on
the embedded brokers:
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">">
<deadLetterStrategy>
<individualDeadLetterStrategy queuePrefix="DLQ."
useQueueForQueueMessages="true" />
</deadLetterStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
I'm running 5.6.0 on brokers and clients with Java 1.6.0.14 on Linux,
embedded and stand alone brokers are persistent.
Any ideas of what I am missing?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Store-and-Forward-with-embedded-brokers-stand-alone-broker-tp4655028.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Store and Forward with embedded brokers + stand alone broker
Posted by ceposta <ch...@gmail.com>.
if you attach your config files i can give it a try on my side
-----
http://www.christianposta.com/blog
--
View this message in context: http://activemq.2283324.n4.nabble.com/Store-and-Forward-with-embedded-brokers-stand-alone-broker-tp4655028p4655431.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Store is locked... waiting 10 seconds
Posted by Torsten Mielke <to...@fusesource.com>.
What version of ActiveMQ is your user using? If its any later version, i.e. 5.5.0 or higher, I suggest switching to KahaDB as the persistence adapter.
Torsten Mielke
torsten@fusesource.com
tmielke@blogspot.com
On Aug 21, 2012, at 8:02 AM, Jamie wrote:
> Hi Everyone
>
> A user is complaining about his application logs being filled up with "Store is locked... waiting 10 seconds". The KahaPersistenceAdapter seems to be having trouble obtaining a lock. I am using an embedded broker, but the broker is only ever started once. I am quite sure of this. The only other scenario I can think of, aside from a possible bug, is that the customer shutdown the application uncleanly and a lock file was left lying around. Has anyone else encountered this situation? Where else should I be looking to solve this. Any ideas?
>
> Much appreciate
>
> Jamie
>
> 2012-08-20 17:10:39 [localhost-startStop-1] INFO
> o.a.activemq.kaha.impl.KahaStore - Kaha Store using data directory
> /var/opt/mailarchiva/ROOT/queue/kaha
> 2012-08-20 17:10:39 [localhost-startStop-1] INFO
> o.a.a.s.k.KahaPersistenceAdapter - Store is locked... waiting 10 seconds
> for the Store to be unlocked.
> 2012-08-20 17:10:49 [localhost-startStop-1] INFO
> o.a.activemq.kaha.impl.KahaStore - Kaha Store using data directory
> /var/opt/mailarchiva/ROOT/queue/kaha
> 2012-08-20 17:10:49 [localhost-startStop-1] INFO
> o.a.a.s.k.KahaPersistenceAdapter - Store is locked... waiting 10 seconds
> for the Store to be unlocked.
> 2012-08-20 17:10:59 [localhost-startStop-1] INFO
> o.a.activemq.kaha.impl.KahaStore - Kaha Store using data directory
> /var/opt/mailarchiva/ROOT/queue/kaha
> 2012-08-20 17:10:59 [localhost-startStop-1] INFO
> o.a.a.s.k.KahaPersistenceAdapter - Store is locked... waiting 10 seconds
> for the Store to be unlocked.
> 2012-08-20 17:11:09 [localhost-startStop-1] INFO
> o.a.activemq.kaha.impl.KahaStore - Kaha Store using data directory
> /var/opt/mailarchiva/ROOT/queue/kaha
> 2012-08-20 17:11:09 [localhost-startStop-1] INFO
> o.a.a.s.k.KahaPersistenceAdapter - Store is locked... waiting 10 seconds
> for the Store to be unlocked.
> 2012-08-20 17:11:19 [localhost-startStop-1] INFO
> o.a.activemq.kaha.impl.KahaStore - Kaha Store using data directory
> /var/opt/mailarchiva/ROOT/queue/kaha
> 2012-08-20 17:11:19 [localhost-startStop-1] INFO
> o.a.a.s.k.KahaPersistenceAdapter - Store is locked... waiting 10 seconds
> for the Store to be unlocked.
>
Store is locked... waiting 10 seconds
Posted by Jamie <ja...@mailarchiva.com>.
Hi Everyone
A user is complaining about his application logs being filled up with
"Store is locked... waiting 10 seconds". The KahaPersistenceAdapter
seems to be having trouble obtaining a lock. I am using an embedded
broker, but the broker is only ever started once. I am quite sure of
this. The only other scenario I can think of, aside from a possible bug,
is that the customer shutdown the application uncleanly and a lock file
was left lying around. Has anyone else encountered this situation? Where
else should I be looking to solve this. Any ideas?
Much appreciate
Jamie
2012-08-20 17:10:39 [localhost-startStop-1] INFO
o.a.activemq.kaha.impl.KahaStore - Kaha Store using data directory
/var/opt/mailarchiva/ROOT/queue/kaha
2012-08-20 17:10:39 [localhost-startStop-1] INFO
o.a.a.s.k.KahaPersistenceAdapter - Store is locked... waiting 10 seconds
for the Store to be unlocked.
2012-08-20 17:10:49 [localhost-startStop-1] INFO
o.a.activemq.kaha.impl.KahaStore - Kaha Store using data directory
/var/opt/mailarchiva/ROOT/queue/kaha
2012-08-20 17:10:49 [localhost-startStop-1] INFO
o.a.a.s.k.KahaPersistenceAdapter - Store is locked... waiting 10 seconds
for the Store to be unlocked.
2012-08-20 17:10:59 [localhost-startStop-1] INFO
o.a.activemq.kaha.impl.KahaStore - Kaha Store using data directory
/var/opt/mailarchiva/ROOT/queue/kaha
2012-08-20 17:10:59 [localhost-startStop-1] INFO
o.a.a.s.k.KahaPersistenceAdapter - Store is locked... waiting 10 seconds
for the Store to be unlocked.
2012-08-20 17:11:09 [localhost-startStop-1] INFO
o.a.activemq.kaha.impl.KahaStore - Kaha Store using data directory
/var/opt/mailarchiva/ROOT/queue/kaha
2012-08-20 17:11:09 [localhost-startStop-1] INFO
o.a.a.s.k.KahaPersistenceAdapter - Store is locked... waiting 10 seconds
for the Store to be unlocked.
2012-08-20 17:11:19 [localhost-startStop-1] INFO
o.a.activemq.kaha.impl.KahaStore - Kaha Store using data directory
/var/opt/mailarchiva/ROOT/queue/kaha
2012-08-20 17:11:19 [localhost-startStop-1] INFO
o.a.a.s.k.KahaPersistenceAdapter - Store is locked... waiting 10 seconds
for the Store to be unlocked.
Re: Store and Forward with embedded brokers + stand alone broker
Posted by pollotek <cl...@gmail.com>.
Torsten, I just tried the configuration you suggested. I made my consumer
Network Connectors duplex and I still don't see any messages making it to
the consumers.
If I connect with JMX to the stand alone broker I can see under
org.apache.activemq.broker.Connection.openwire my 3 producers connected, I
can identify the IP addresses of the servers.
Under org.apache.activemq.broker.NetworkBridge I see my two duplex clients,
I see in the connection attributes CreatedByDuplex=true. I also see under
org.apache.activemq.brokerQueue.<name-of-my-queue>.Attributes.Subscriptions
my consumers as subscribers.
I implemented listeners for the queue advisories on the embedded brokers.
After I place a new message on the producer side I see the size of the queue
growing on the producer embedded broker but I don't see dequeueCount or
inflightCount ever changing from 0, only size changes. On the stand alone
broker queue stats I don't see EnqueueCount, DispatchCount, InFlightCount or
QueueSize ever changing from 0 either.
I also see on the stand alone broker's log messages like this one, one per
duplex consumer:
2012-08-20 21:40:03,574 | INFO | Started responder end of duplex bridge
connection_to_nob-10.17.163.168-8080@ID:mc02-48630-1345498803060-0:1 |
org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
tcp:///10.17.163.168:56989
2012-08-20 21:40:03,576 | INFO | Network connection between vm://broker#2
and tcp:///10.17.163.168:56989(localhost) has been established. |
org.apache.activemq.network.DemandForwardingBridgeSupport |
StartLocalBridge: localBroker=vm://broker#2
Do you guys have any other ideas? Would it be handy to submit configuration
files or log files on a specific log level to help solve this issue?
It's very strange that if I test the topology you suggested to test first,
to have only embedded brokers on the producer side but the consumers
connected directly messages actually flow. But if they both have embedded
brokers messages just don't flow to the consumers.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Store-and-Forward-with-embedded-brokers-stand-alone-broker-tp4655028p4655340.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Store and Forward with embedded brokers + stand alone broker
Posted by pollotek <cl...@gmail.com>.
Hi torsten, in-line comments.
On Mon, Aug 13, 2012 at 12:33 AM, Torsten Mielke-2 [via ActiveMQ] <
ml-node+s2283324n4655036h72@n4.nabble.com> wrote:
> Hello,
>
> Some answers in-line.
>
>
> On Aug 11, 2012, at 4:20 AM, pollotek wrote:
>
> > I have this web application that currently connects to a stand alone
> broker
> > to place messages and consumer only nodes pick up and process the
> messages.
> > I want to move this to have embedded brokers on each web app and each
> > consumer and those embedded brokers should forward their messages to a
> stand
> > alone broker that will sit in the middle just like it does today.
> >
> > producer app server -> embedded broker -> stand alone broker -> embedded
> > broker -> consumer app server
> >
> > This will allow messages to be produced and queued up locally without
> > blocking and have the messages forwarded asynchronously.
>
> True but at the expense of a more complex topology. If you have a higher
> number of producers and consumer, you will end up with a fairly huge
> hub-and-spoke broker network topology. Do you really expect your JMS
> clients to be disconnected from the standalone broker that often that you
> want to additionally run with embedded brokers?
>
What would be the implications of a more complex topology? I don't want
messages production calls to block, I read I could use timeouts but if my
call to place a message on the stand alone broker times out I have to
handle the case which complicates my code. If the message gets queued up
locally by the embedded broker my code becomes much simpler.
I also read in the AMQ documentation about the existence of some buffering
mechanism so messages are placed in a local buffer if the stand alone
broker is down but that mechanism is not persistent and I read comments
where embedded broker was the reliable way to implement this functionality.
>
>
>
> > As I understand the
> > only thing to do here is to declare the embedded brokers on both app
> servers
> > and point their network connectors to the stand alone broker. Messages
> are
> > supposed to be stored in the embedded broker and if there are
> subscribers to
> > the queue on the stand alone broker side messages will be forwarded.
>
> Correct. However the network connector from your consumers embedded broker
> to the standalone broker needs to be duplex, as otherwise msgs can only
> from from that embedded broker to the standalone broker.
>
> >
> > So I configured my embedded brokers with these parameters (this is the
> same
> > configuration on both, consumers and producers) :
> >
> > <amq:transportConnectors>
> > <amq:transportConnector uri="vm://localhost?async=true" />
> > </amq:transportConnectors>
> >
> > <amq:networkConnectors>
> > <amq:networkConnector
> > name="connection_to_nob-${activemq.brokerid}"
> >
> >
> uri="static:failover:(tcp://broker01:61616,tcp://broker02:61616)?randomize=false&maxReconnectAttempts=0"/>
>
> > </amq:networkConnectors>
>
> Add duplex=true to the network connectors of your consumer embedded
> brokers as mentioned above.
>
>
> > The stand alone broker of course is configured to receive connections on
> > 61616. I start up my application and make sure with JMX all consumers
> and
> > producers are connected to the same broker (Master-slave shared file
> system)
> > and everything looks ok so far. I create some messages for queues I have
> > consumers for declared in the consumer side only.
>
> To get up and running and eliminate possible errors, perhaps try without
> master/slave first.
> >
> > I point my producer and consumer app servers to connect to
> vm://localhost. I
> > never see a single message leave the embedded broker on the producers
> side.
> > I can see with JMX the stand alone broker doesn't create the queues
> where my
> > process is placing new messages.
>
> Could be done to the fact that some of your network connectors needs to be
> duplex.
> For testing try to connect with a consumer to the standalone broker first
> and see if msgs get passed from the producer's embedded broker to the
> standalone broker and to your directly connected consumer.
> If they do, then your producer side configuration is correct. Then check
> the consumers config.
>
I actually ran this test already successfully with embedded brokers on the
producer side and direct connections to the stand alone broker on the
consumer side.
>
> >
> > As I understand from what I read I don't need to specifically create the
> > queues to get messages forwarded to in the stand alone broker, they're
> > supposed to be automatically created.
>
> Correct, destinations get created automatically by the brokers once a
> producer or consumer is connected.
>
>
> > I don't have anything special on my
> > stand alone broker other than this destination Policy which I also have
> on
> > the embedded brokers:
> >
> > <destinationPolicy>
> > <policyMap>
> > <policyEntries>
> >
> > <policyEntry queue=">">
> > <deadLetterStrategy>
> >
> > <individualDeadLetterStrategy
> queuePrefix="DLQ."
> > useQueueForQueueMessages="true" />
> > </deadLetterStrategy>
> > </policyEntry>
> > </policyEntries>
> > </policyMap>
> > </destinationPolicy>
> >
>
> That config is fine and would not cause your problem.
> See my above points and also verify that you did not disable support for
> advisory messages explicitly in your broker configurations. These are
> needed in a broker network to share information about connected JMS
> clients.
> http://activemq.apache.org/advisory-message.html
>
>
>
> > I'm running 5.6.0 on brokers and clients with Java 1.6.0.14 on Linux,
> > embedded and stand alone brokers are persistent.
> >
> > Any ideas of what I am missing?
> >
>
>
> Hope this helps a bit.
>
>
> Torsten Mielke
> [hidden email] <http://user/SendEmail.jtp?type=node&node=4655036&i=0>
> [hidden email] <http://user/SendEmail.jtp?type=node&node=4655036&i=1>
>
>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://activemq.2283324.n4.nabble.com/Store-and-Forward-with-embedded-brokers-stand-alone-broker-tp4655028p4655036.html
> To unsubscribe from Store and Forward with embedded brokers + stand alone
> broker, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4655028&code=Y2xhdWRpby5zYW50YW5hQGdtYWlsLmNvbXw0NjU1MDI4fC0xNTA0MDM5ODky>
> .
> NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
I won't be able to run this test for a few days. I'll let you know the
results.
Thanks for the quick and detailed response!
Claudio
--
View this message in context: http://activemq.2283324.n4.nabble.com/Store-and-Forward-with-embedded-brokers-stand-alone-broker-tp4655028p4655076.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Store and Forward with embedded brokers + stand alone broker
Posted by Torsten Mielke <to...@fusesource.com>.
Hello,
Some answers in-line.
On Aug 11, 2012, at 4:20 AM, pollotek wrote:
> I have this web application that currently connects to a stand alone broker
> to place messages and consumer only nodes pick up and process the messages.
> I want to move this to have embedded brokers on each web app and each
> consumer and those embedded brokers should forward their messages to a stand
> alone broker that will sit in the middle just like it does today.
>
> producer app server -> embedded broker -> stand alone broker -> embedded
> broker -> consumer app server
>
> This will allow messages to be produced and queued up locally without
> blocking and have the messages forwarded asynchronously.
True but at the expense of a more complex topology. If you have a higher number of producers and consumer, you will end up with a fairly huge hub-and-spoke broker network topology. Do you really expect your JMS clients to be disconnected from the standalone broker that often that you want to additionally run with embedded brokers?
> As I understand the
> only thing to do here is to declare the embedded brokers on both app servers
> and point their network connectors to the stand alone broker. Messages are
> supposed to be stored in the embedded broker and if there are subscribers to
> the queue on the stand alone broker side messages will be forwarded.
Correct. However the network connector from your consumers embedded broker to the standalone broker needs to be duplex, as otherwise msgs can only from from that embedded broker to the standalone broker.
>
> So I configured my embedded brokers with these parameters (this is the same
> configuration on both, consumers and producers) :
>
> <amq:transportConnectors>
> <amq:transportConnector uri="vm://localhost?async=true" />
> </amq:transportConnectors>
>
> <amq:networkConnectors>
> <amq:networkConnector
> name="connection_to_nob-${activemq.brokerid}"
>
> uri="static:failover:(tcp://broker01:61616,tcp://broker02:61616)?randomize=false&maxReconnectAttempts=0"/>
> </amq:networkConnectors>
Add duplex=true to the network connectors of your consumer embedded brokers as mentioned above.
> The stand alone broker of course is configured to receive connections on
> 61616. I start up my application and make sure with JMX all consumers and
> producers are connected to the same broker (Master-slave shared file system)
> and everything looks ok so far. I create some messages for queues I have
> consumers for declared in the consumer side only.
To get up and running and eliminate possible errors, perhaps try without master/slave first.
>
> I point my producer and consumer app servers to connect to vm://localhost. I
> never see a single message leave the embedded broker on the producers side.
> I can see with JMX the stand alone broker doesn't create the queues where my
> process is placing new messages.
Could be done to the fact that some of your network connectors needs to be duplex.
For testing try to connect with a consumer to the standalone broker first and see if msgs get passed from the producer's embedded broker to the standalone broker and to your directly connected consumer.
If they do, then your producer side configuration is correct. Then check the consumers config.
>
> As I understand from what I read I don't need to specifically create the
> queues to get messages forwarded to in the stand alone broker, they're
> supposed to be automatically created.
Correct, destinations get created automatically by the brokers once a producer or consumer is connected.
> I don't have anything special on my
> stand alone broker other than this destination Policy which I also have on
> the embedded brokers:
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
>
> <policyEntry queue=">">
> <deadLetterStrategy>
>
> <individualDeadLetterStrategy queuePrefix="DLQ."
> useQueueForQueueMessages="true" />
> </deadLetterStrategy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
That config is fine and would not cause your problem.
See my above points and also verify that you did not disable support for advisory messages explicitly in your broker configurations. These are needed in a broker network to share information about connected JMS clients.
http://activemq.apache.org/advisory-message.html
> I'm running 5.6.0 on brokers and clients with Java 1.6.0.14 on Linux,
> embedded and stand alone brokers are persistent.
>
> Any ideas of what I am missing?
>
Hope this helps a bit.
Torsten Mielke
torsten@fusesource.com
tmielke@blogspot.com