You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by magellings <ma...@qg.com> on 2009/04/14 00:18:08 UTC
NMS 1.1 and failover protocol slowness
Hello. Has anyone used the 1.1 binaries for NMS and NMS.ActiveMQ from trunk
and the failover: protocol?
I'm experiencing a lot of slowness and I've found at least one comment (on
item in issue tracker) stating the same.
http://issues.apache.org/activemq/browse/AMQNET-26?focusedCommentId=47183&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_47183
"Second, the recent changes to Apache.NMS.ActiveMQ's file
OpenWire/OpenWireFormat.cs create massive delays on my system. If I revert
that file back to the version 693666 (09-Sep-2008) then all my code starts
working without delays. It seems that the action of writing to the bus
(using Topics in my case) is delayed 120 seconds or so. Each write seems to
be delayed 15 seconds. Since startup involves several writes the initialize
the wire format, I suspect that a timeout variable is set to 0 (zero) which
does not allow a thread-swap to take place. So it hangs. I will create a new
issue for this problem.
Please test and give feedback. If it is positive, then Jim can commit the
change to the repository."
I've also noticed the Spring Messaging framework stalls when master and
slave brokers switch using the failover protocol.
--
View this message in context: http://www.nabble.com/NMS-1.1-and-failover-protocol-slowness-tp23028265p23028265.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: NMS 1.1 and failover protocol slowness
Posted by magellings <ma...@qg.com>.
I have created an enhancement item in the NMS jira. A patch is included to
asynchronously connect with the failover protocol. This will lessen the
impact of the socket connection timeout from the client perspective. Patch
included.
http://issues.apache.org/activemq/browse/AMQNET-159
magellings wrote:
>
> I may have found a work around but I don't know the full implications of
> it.
>
> Spring allows one to initialize synchronization. Doing this along with
> reusing a ConnectionFactory will cache the session in which you don't take
> the hit of a connection attempt to the slave broker when each message is
> sent. If anyone knows any additional information about this
> Synchronization Manager please let me know.
>
> TransactionSynchronizationManager.InitSynchronization();
>
>
> var connectionFactory = new ConnectionFactory(_uri);
> connectionFactory.UserName = "frontend";
> connectionFactory.Password = "pizz@";
>
>
> magellings wrote:
>>
>> I believe I figured out the performance problem. Currently we are using
>> JDBC master/slave.
>>
>> The major performance problem is the socket connection is timing out on
>> whichever tcp address in the failover address is slave.
>>
>> Example:
>>
>> private static Uri _uri = new
>> Uri("failover:(tcp://localhost:10083,tcp://WAMQDEV1.qg.com:10083)?Randomize=false");
>>
>> If "localhost" is running the slave broker, then it takes roughly 20-25s
>> on each establishment of a session. This is because the first connection
>> attempt to "localhost" takes this long to timeout. It then connects to
>> "WAMQDEV1", the master, quickly. If "localhost" is the master (the other
>> way around) than performance is fine as "localhost" is tried first and a
>> connection to "WAMQDEV1" is never attempted.
>>
>> The problem is partially related to how the Spring NMS framework
>> NmsTemplate connects as it establishes a new session on each message
>> sent. So the performance hit is taken on each message sent. This isn't
>> necessarily wrong though as we need to ensure we are sending to the
>> master broker at the time of the message sending.
>>
>> A second performance problem is the sheer fact that it takes roughly "1"
>> second for each message sent now. This is without utilizing the failover
>> protocol functionality. This is a pretty large performance hit compared
>> to NMS version 1.0. For example, now if 100 messages are sent the
>> elapsed time is approximately 100 seconds. I have yet to profile
>> attempting to figure out why it is slower sending a message.
>>
>> In regards to the failover protocol performance hit, if it is okay I will
>> attempt to fix and will submit a patch. I believe it should be as simple
>> as tracking the successfully connected address and enhancing the logic to
>> always try the address that previously connected successfully first. If
>> it fails than the successfully connected address is updated to the
>> address of the new master broker and the process repeats.
>>
>>
>> magellings wrote:
>>>
>>> Hello. Has anyone used the 1.1 binaries for NMS and NMS.ActiveMQ from
>>> trunk and the failover: protocol?
>>>
>>> I'm experiencing a lot of slowness and I've found at least one comment
>>> (on item in issue tracker) stating the same.
>>>
>>> http://issues.apache.org/activemq/browse/AMQNET-26?focusedCommentId=47183&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_47183
>>>
>>> "Second, the recent changes to Apache.NMS.ActiveMQ's file
>>> OpenWire/OpenWireFormat.cs create massive delays on my system. If I
>>> revert that file back to the version 693666 (09-Sep-2008) then all my
>>> code starts working without delays. It seems that the action of writing
>>> to the bus (using Topics in my case) is delayed 120 seconds or so. Each
>>> write seems to be delayed 15 seconds. Since startup involves several
>>> writes the initialize the wire format, I suspect that a timeout variable
>>> is set to 0 (zero) which does not allow a thread-swap to take place. So
>>> it hangs. I will create a new issue for this problem.
>>>
>>> Please test and give feedback. If it is positive, then Jim can commit
>>> the change to the repository."
>>>
>>> I've also noticed the Spring Messaging framework stalls when master and
>>> slave brokers switch using the failover protocol.
>>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/NMS-1.1-and-failover-protocol-slowness-tp23028265p23224896.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: NMS 1.1 and failover protocol slowness
Posted by magellings <ma...@qg.com>.
I may have found a work around but I don't know the full implications of it.
Spring allows one to initialize synchronization. Doing this along with
reusing a ConnectionFactory will cache the session in which you don't take
the hit of a connection attempt to the slave broker when each message is
sent. If anyone knows any additional information about this Synchronization
Manager please let me know.
TransactionSynchronizationManager.InitSynchronization();
var connectionFactory = new ConnectionFactory(_uri);
connectionFactory.UserName = "frontend";
connectionFactory.Password = "pizz@";
magellings wrote:
>
> I believe I figured out the performance problem. Currently we are using
> JDBC master/slave.
>
> The major performance problem is the socket connection is timing out on
> whichever tcp address in the failover address is slave.
>
> Example:
>
> private static Uri _uri = new
> Uri("failover:(tcp://localhost:10083,tcp://WAMQDEV1.qg.com:10083)?Randomize=false");
>
> If "localhost" is running the slave broker, then it takes roughly 20-25s
> on each establishment of a session. This is because the first connection
> attempt to "localhost" takes this long to timeout. It then connects to
> "WAMQDEV1", the master, quickly. If "localhost" is the master (the other
> way around) than performance is fine as "localhost" is tried first and a
> connection to "WAMQDEV1" is never attempted.
>
> The problem is partially related to how the Spring NMS framework
> NmsTemplate connects as it establishes a new session on each message sent.
> So the performance hit is taken on each message sent. This isn't
> necessarily wrong though as we need to ensure we are sending to the master
> broker at the time of the message sending.
>
> A second performance problem is the sheer fact that it takes roughly "1"
> second for each message sent now. This is without utilizing the failover
> protocol functionality. This is a pretty large performance hit compared
> to NMS version 1.0. For example, now if 100 messages are sent the elapsed
> time is approximately 100 seconds. I have yet to profile attempting to
> figure out why it is slower sending a message.
>
> In regards to the failover protocol performance hit, if it is okay I will
> attempt to fix and will submit a patch. I believe it should be as simple
> as tracking the successfully connected address and enhancing the logic to
> always try the address that previously connected successfully first. If
> it fails than the successfully connected address is updated to the address
> of the new master broker and the process repeats.
>
>
> magellings wrote:
>>
>> Hello. Has anyone used the 1.1 binaries for NMS and NMS.ActiveMQ from
>> trunk and the failover: protocol?
>>
>> I'm experiencing a lot of slowness and I've found at least one comment
>> (on item in issue tracker) stating the same.
>>
>> http://issues.apache.org/activemq/browse/AMQNET-26?focusedCommentId=47183&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_47183
>>
>> "Second, the recent changes to Apache.NMS.ActiveMQ's file
>> OpenWire/OpenWireFormat.cs create massive delays on my system. If I
>> revert that file back to the version 693666 (09-Sep-2008) then all my
>> code starts working without delays. It seems that the action of writing
>> to the bus (using Topics in my case) is delayed 120 seconds or so. Each
>> write seems to be delayed 15 seconds. Since startup involves several
>> writes the initialize the wire format, I suspect that a timeout variable
>> is set to 0 (zero) which does not allow a thread-swap to take place. So
>> it hangs. I will create a new issue for this problem.
>>
>> Please test and give feedback. If it is positive, then Jim can commit the
>> change to the repository."
>>
>> I've also noticed the Spring Messaging framework stalls when master and
>> slave brokers switch using the failover protocol.
>>
>
>
--
View this message in context: http://www.nabble.com/NMS-1.1-and-failover-protocol-slowness-tp23028265p23107014.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: NMS 1.1 and failover protocol slowness
Posted by magellings <ma...@qg.com>.
I believe I figured out the performance problem. Currently we are using JDBC
master/slave.
The major performance problem is the socket connection is timing out on
whichever tcp address in the failover address is slave.
Example:
private static Uri _uri = new
Uri("failover:(tcp://localhost:10083,tcp://WAMQDEV1.qg.com:10083)?Randomize=false");
If "localhost" is running the slave broker, then it takes roughly 20-25s on
each establishment of a session. This is because the first connection
attempt to "localhost" takes this long to timeout. It then connects to
"WAMQDEV1", the master, quickly. If "localhost" is the master (the other
way around) than performance is fine as "localhost" is tried first and a
connection to "WAMQDEV1" is never attempted.
The problem is partially related to how the Spring NMS framework NmsTemplate
connects as it establishes a new session on each message sent. So the
performance hit is taken on each message sent. This isn't necessarily wrong
though as we need to ensure we are sending to the master broker at the time
of the message sending.
A second performance problem is the sheer fact that it takes roughly "1"
second for each message sent now. This is without utilizing the failover
protocol functionality. This is a pretty large performance hit compared to
NMS version 1.0. For example, now if 100 messages are sent the elapsed time
is approximately 100 seconds. I have yet to profile attempting to figure
out why it is slower sending a message.
In regards to the failover protocol performance hit, if it is okay I will
attempt to fix and will submit a patch. I believe it should be as simple as
tracking the successfully connected address and enhancing the logic to
always try the address that previously connected successfully first. If it
fails than the successfully connected address is updated to the address of
the new master broker and the process repeats.
magellings wrote:
>
> Hello. Has anyone used the 1.1 binaries for NMS and NMS.ActiveMQ from
> trunk and the failover: protocol?
>
> I'm experiencing a lot of slowness and I've found at least one comment (on
> item in issue tracker) stating the same.
>
> http://issues.apache.org/activemq/browse/AMQNET-26?focusedCommentId=47183&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_47183
>
> "Second, the recent changes to Apache.NMS.ActiveMQ's file
> OpenWire/OpenWireFormat.cs create massive delays on my system. If I revert
> that file back to the version 693666 (09-Sep-2008) then all my code starts
> working without delays. It seems that the action of writing to the bus
> (using Topics in my case) is delayed 120 seconds or so. Each write seems
> to be delayed 15 seconds. Since startup involves several writes the
> initialize the wire format, I suspect that a timeout variable is set to 0
> (zero) which does not allow a thread-swap to take place. So it hangs. I
> will create a new issue for this problem.
>
> Please test and give feedback. If it is positive, then Jim can commit the
> change to the repository."
>
> I've also noticed the Spring Messaging framework stalls when master and
> slave brokers switch using the failover protocol.
>
--
View this message in context: http://www.nabble.com/NMS-1.1-and-failover-protocol-slowness-tp23028265p23103499.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.