You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Arthur Naseef (JIRA)" <ji...@apache.org> on 2011/05/25 23:11:47 UTC
[jira] [Created] (AMQ-3336) Temporary Destination errors on H/A
failover in broker network with Failover transport
Temporary Destination errors on H/A failover in broker network with Failover transport
--------------------------------------------------------------------------------------
Key: AMQ-3336
URL: https://issues.apache.org/jira/browse/AMQ-3336
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.4.2
Environment: Client using failover transport
Network of brokers with failover transports using maxReconnectAttempts=1
Temporary Queues and Topics
5.4.2 with patches
Reporter: Arthur Naseef
Priority: Minor
Based on the following page, the failover transport should handle Temporary destinations transparent to clients after a failover:
{code}
http://activemq.apache.org/how-do-i-configure-automatic-reconnection.html
{code}
However, I have a test in which a producer to a Temporary Destination receives the following error on a temporary destination created before a failover, when producing after the failover (message truncated):
{code}
javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID: ...
{code}
Here's the test steps:
# create broker network (using 3 nodes in this test, but 2 should suffice)
#* each broker connects to the other 2 with duplex network connectors
#* failover transport is used (custom "single" transport also used to test)
#* H/A pairs; each NODE consits of 2 brokers using shared fileystem locking
# start an echo service connected to node "2"
#* service listens on a permanent queue
#* replyTo header used to send a copy of message back to the sender
#* on request, create producer to the replyTo destination, send response, close producer
#* using failover transport
# start an echo client connected to node "1"
#* a temporary Queue or Topic (2 separate tests) is created
#* client sends requests to the echo service with replyTo of the temp dest
#* requests sent at regular intervals asynchronous of response processing
#* multiple requests sent in sequence
# after a short delay, node "1" is forced to fail over
#* BrokerService stop method called on the active broker in the node
#* failover completes succesfully
# next request by the echo client to the service result in the following error (message truncated)
{code}
javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID: ...
{code}
Can the failover transport and broker support transparent recovery of the temporary destination by a producer on a different broker/connection?
Note that this sequence works fine with queues, as-expected. Topics with non-durable subscriptions lose messages, but that's likely expected due to the consumer being off the bus for a period of time.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3336) Temporary Destination errors on H/A
failover in broker network with Failover transport
Posted by "Arthur Naseef (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039774#comment-13039774 ]
Arthur Naseef commented on AMQ-3336:
------------------------------------
I see. I'll try the advisory setting - that sounds good.
For abortive, the primary purpose is H/A testing of a failover, although I can test this use case with that method... I'll look at that.
Thanks!
> Temporary Destination errors on H/A failover in broker network with Failover transport
> --------------------------------------------------------------------------------------
>
> Key: AMQ-3336
> URL: https://issues.apache.org/jira/browse/AMQ-3336
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Client using failover transport
> Network of brokers with failover transports using maxReconnectAttempts=1
> Temporary Queues and Topics
> 5.4.2 with patches
> Reporter: Arthur Naseef
> Priority: Minor
>
> Based on the following page, the failover transport should handle Temporary destinations transparent to clients after a failover:
> {code}
> http://activemq.apache.org/how-do-i-configure-automatic-reconnection.html
> {code}
> However, I have a test in which a producer to a Temporary Destination receives the following error on a temporary destination created before a failover, when producing after the failover (message truncated):
> {code}
> javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID: ...
> {code}
> Here's the test steps:
> # create broker network (using 3 nodes in this test, but 2 should suffice)
> #* each broker connects to the other 2 with duplex network connectors
> #* failover transport is used (custom "single" transport also used to test)
> #* H/A pairs; each NODE consits of 2 brokers using shared fileystem locking
> # start an echo service connected to node "2"
> #* service listens on a permanent queue
> #* replyTo header used to send a copy of message back to the sender
> #* on request, create producer to the replyTo destination, send response, close producer
> #* using failover transport
> # start an echo client connected to node "1"
> #* a temporary Queue or Topic (2 separate tests) is created
> #* client sends requests to the echo service with replyTo of the temp dest
> #* requests sent at regular intervals asynchronous of response processing
> #* multiple requests sent in sequence
> # after a short delay, node "1" is forced to fail over
> #* BrokerService stop method called on the active broker in the node
> #* failover completes succesfully
> # next request by the echo client to the service result in the following error (message truncated)
> {code}
> javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID: ...
> {code}
> Can the failover transport and broker support transparent recovery of the temporary destination by a producer on a different broker/connection?
> Note that this sequence works fine with queues, as-expected. Topics with non-durable subscriptions lose messages, but that's likely expected due to the consumer being off the bus for a period of time.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3336) Temporary Destination errors on H/A
failover in broker network with Failover transport
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039624#comment-13039624 ]
Gary Tully commented on AMQ-3336:
---------------------------------
the failover transport will recreate temp destinations but an activemq connection tracks a temp destination deleted advisory.
When the broker is shutdown via stop, the temp dest is deleted as part of the connection close and the advisory fires so the connection sees the destination as deleted.
You need to configure the activemq connection to not track temp advisories:
jms.watchTopicAdvisories=false as a url param on the client brokerURL, or via the corresponding attribute on the connection or connection factory.
> Temporary Destination errors on H/A failover in broker network with Failover transport
> --------------------------------------------------------------------------------------
>
> Key: AMQ-3336
> URL: https://issues.apache.org/jira/browse/AMQ-3336
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Client using failover transport
> Network of brokers with failover transports using maxReconnectAttempts=1
> Temporary Queues and Topics
> 5.4.2 with patches
> Reporter: Arthur Naseef
> Priority: Minor
>
> Based on the following page, the failover transport should handle Temporary destinations transparent to clients after a failover:
> {code}
> http://activemq.apache.org/how-do-i-configure-automatic-reconnection.html
> {code}
> However, I have a test in which a producer to a Temporary Destination receives the following error on a temporary destination created before a failover, when producing after the failover (message truncated):
> {code}
> javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID: ...
> {code}
> Here's the test steps:
> # create broker network (using 3 nodes in this test, but 2 should suffice)
> #* each broker connects to the other 2 with duplex network connectors
> #* failover transport is used (custom "single" transport also used to test)
> #* H/A pairs; each NODE consits of 2 brokers using shared fileystem locking
> # start an echo service connected to node "2"
> #* service listens on a permanent queue
> #* replyTo header used to send a copy of message back to the sender
> #* on request, create producer to the replyTo destination, send response, close producer
> #* using failover transport
> # start an echo client connected to node "1"
> #* a temporary Queue or Topic (2 separate tests) is created
> #* client sends requests to the echo service with replyTo of the temp dest
> #* requests sent at regular intervals asynchronous of response processing
> #* multiple requests sent in sequence
> # after a short delay, node "1" is forced to fail over
> #* BrokerService stop method called on the active broker in the node
> #* failover completes succesfully
> # next request by the echo client to the service result in the following error (message truncated)
> {code}
> javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID: ...
> {code}
> Can the failover transport and broker support transparent recovery of the temporary destination by a producer on a different broker/connection?
> Note that this sequence works fine with queues, as-expected. Topics with non-durable subscriptions lose messages, but that's likely expected due to the consumer being off the bus for a period of time.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3336) Temporary Destination errors on H/A
failover in broker network with Failover transport
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039767#comment-13039767 ]
Gary Tully commented on AMQ-3336:
---------------------------------
On the advisory, in an ActiveMQConnection, the advisory is only used for temp deletion detection as an optimization to save on a round trip to the broker. So the impact will be limited to just your use case. You still leave advisorySupport=true for the broker.
To simulate an abortive broker shutdown for clients I use the org.apache.activemq.util.SocketProxy test class from activemq-core, that easily allows the connection to be aborted between the client and broker. See usage is something like
org.apache.activemq.usecases.BrokerQueueNetworkWithDisconnectTest
> Temporary Destination errors on H/A failover in broker network with Failover transport
> --------------------------------------------------------------------------------------
>
> Key: AMQ-3336
> URL: https://issues.apache.org/jira/browse/AMQ-3336
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Client using failover transport
> Network of brokers with failover transports using maxReconnectAttempts=1
> Temporary Queues and Topics
> 5.4.2 with patches
> Reporter: Arthur Naseef
> Priority: Minor
>
> Based on the following page, the failover transport should handle Temporary destinations transparent to clients after a failover:
> {code}
> http://activemq.apache.org/how-do-i-configure-automatic-reconnection.html
> {code}
> However, I have a test in which a producer to a Temporary Destination receives the following error on a temporary destination created before a failover, when producing after the failover (message truncated):
> {code}
> javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID: ...
> {code}
> Here's the test steps:
> # create broker network (using 3 nodes in this test, but 2 should suffice)
> #* each broker connects to the other 2 with duplex network connectors
> #* failover transport is used (custom "single" transport also used to test)
> #* H/A pairs; each NODE consits of 2 brokers using shared fileystem locking
> # start an echo service connected to node "2"
> #* service listens on a permanent queue
> #* replyTo header used to send a copy of message back to the sender
> #* on request, create producer to the replyTo destination, send response, close producer
> #* using failover transport
> # start an echo client connected to node "1"
> #* a temporary Queue or Topic (2 separate tests) is created
> #* client sends requests to the echo service with replyTo of the temp dest
> #* requests sent at regular intervals asynchronous of response processing
> #* multiple requests sent in sequence
> # after a short delay, node "1" is forced to fail over
> #* BrokerService stop method called on the active broker in the node
> #* failover completes succesfully
> # next request by the echo client to the service result in the following error (message truncated)
> {code}
> javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID: ...
> {code}
> Can the failover transport and broker support transparent recovery of the temporary destination by a producer on a different broker/connection?
> Note that this sequence works fine with queues, as-expected. Topics with non-durable subscriptions lose messages, but that's likely expected due to the consumer being off the bus for a period of time.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3336) Temporary Destination errors on H/A
failover in broker network with Failover transport
Posted by "Arthur Naseef (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039743#comment-13039743 ]
Arthur Naseef commented on AMQ-3336:
------------------------------------
Disabling the advisories will lead to other issues for our network. Perhaps the right answer here is to distinguish a clean broker shutdown from an abrupt one. On a clean shutdown, a formal process can be used to eliminate, or minimize, the impact on clients of temporary destinations.
I'll test abrupt broker termination. Any tips on doing so in a JUnit with multiple brokers (BrokerService instances) would be appreciated.
> Temporary Destination errors on H/A failover in broker network with Failover transport
> --------------------------------------------------------------------------------------
>
> Key: AMQ-3336
> URL: https://issues.apache.org/jira/browse/AMQ-3336
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.4.2
> Environment: Client using failover transport
> Network of brokers with failover transports using maxReconnectAttempts=1
> Temporary Queues and Topics
> 5.4.2 with patches
> Reporter: Arthur Naseef
> Priority: Minor
>
> Based on the following page, the failover transport should handle Temporary destinations transparent to clients after a failover:
> {code}
> http://activemq.apache.org/how-do-i-configure-automatic-reconnection.html
> {code}
> However, I have a test in which a producer to a Temporary Destination receives the following error on a temporary destination created before a failover, when producing after the failover (message truncated):
> {code}
> javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID: ...
> {code}
> Here's the test steps:
> # create broker network (using 3 nodes in this test, but 2 should suffice)
> #* each broker connects to the other 2 with duplex network connectors
> #* failover transport is used (custom "single" transport also used to test)
> #* H/A pairs; each NODE consits of 2 brokers using shared fileystem locking
> # start an echo service connected to node "2"
> #* service listens on a permanent queue
> #* replyTo header used to send a copy of message back to the sender
> #* on request, create producer to the replyTo destination, send response, close producer
> #* using failover transport
> # start an echo client connected to node "1"
> #* a temporary Queue or Topic (2 separate tests) is created
> #* client sends requests to the echo service with replyTo of the temp dest
> #* requests sent at regular intervals asynchronous of response processing
> #* multiple requests sent in sequence
> # after a short delay, node "1" is forced to fail over
> #* BrokerService stop method called on the active broker in the node
> #* failover completes succesfully
> # next request by the echo client to the service result in the following error (message truncated)
> {code}
> javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID: ...
> {code}
> Can the failover transport and broker support transparent recovery of the temporary destination by a producer on a different broker/connection?
> Note that this sequence works fine with queues, as-expected. Topics with non-durable subscriptions lose messages, but that's likely expected due to the consumer being off the bus for a period of time.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira