You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Rob Lugt (JIRA)" <ji...@apache.org> on 2006/10/18 10:26:18 UTC
[jira] Created: (AMQ-988) Thread synchronization error in
TcpTransport
Thread synchronization error in TcpTransport
--------------------------------------------
Key: AMQ-988
URL: https://issues.apache.org/activemq/browse/AMQ-988
Project: ActiveMQ
Issue Type: Bug
Components: NMS (C# client)
Affects Versions: 4.0.2
Environment: Windows
Reporter: Rob Lugt
I have a problem where my C# client application crashes when placed under load. It's taken a while to get to the bottom of it, but I believe I have identified the problem and luckily there's a simple solution.
The AMQ .Net client uses a common pattern where a full-duplex TCP/IP connection is established with the broker, and the client uses two threads to concurrently read and write data to/from the underlying socket - one thread reading from a Reader object and the other writing to a Writer object.
The TcpTransport.Start() method contains the following:-
NetworkStream networkStream = new NetworkStream(socket);
socketWriter = new OpenWireBinaryWriter(networkStream);
socketReader = new OpenWireBinaryReader(networkStream);
This pattern is explicitly allowed in Java and Win32 applications, but .Net is a little vague on the subject. The Microsoft documentation [1] suggests use of the asynchronous read/write calls for multithreaded applications, but that would significantly complicate the C# client which relies on blocking stream behaviour. On the same doc page
Microsoft does specifiy that the Socket class is thread-safe, which I take to mean that concurrent read/write calls are permitted, however the same is not true of NetworkStream [2].
Perhaps it would be reasonable to expect NetworkStream to have no internal state other than the Socket it contains, but apparently this is not the case because I've identified that it is a corrupt NetworkStream which is causing my application to crash under load. After some experimentation and a fair amount of load testing, I think the solution is a simple change to the TcpTransport.start() method to use a separate NetworkStream for input and output operations. e.g. :-
NetworkStream inputNetworkStream = new NetworkStream(socket);
NetworkStream outputNetworkStream = new NetworkStream(socket);
socketWriter = new OpenWireBinaryWriter(inputNetworkStream);
socketReader = new OpenWireBinaryReader(outputNetworkStream);
This works for me and my test client has now been running for 16 hours without crashing (before it would rarely last longer than 10 minutes).
Regards
Rob Lugt
[1] http://msdn2.microsoft.com/en-us/library/system.net.sockets.socket.aspx
[2] http://msdn2.microsoft.com/en-us/library/system.net.sockets.networkstream.aspx
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (AMQ-988) Thread synchronization error in
TcpTransport
Posted by "james strachan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-988?page=all ]
james strachan resolved AMQ-988.
--------------------------------
Resolution: Fixed
Patch applied - thanks!
> Thread synchronization error in TcpTransport
> --------------------------------------------
>
> Key: AMQ-988
> URL: https://issues.apache.org/activemq/browse/AMQ-988
> Project: ActiveMQ
> Issue Type: Bug
> Components: NMS (C# client)
> Affects Versions: 4.0.2
> Environment: Windows
> Reporter: Rob Lugt
> Assigned To: james strachan
> Fix For: 4.1
>
> Attachments: amq988-patch.txt
>
>
> I have a problem where my C# client application crashes when placed under load. It's taken a while to get to the bottom of it, but I believe I have identified the problem and luckily there's a simple solution.
> The AMQ .Net client uses a common pattern where a full-duplex TCP/IP connection is established with the broker, and the client uses two threads to concurrently read and write data to/from the underlying socket - one thread reading from a Reader object and the other writing to a Writer object.
> The TcpTransport.Start() method contains the following:-
> NetworkStream networkStream = new NetworkStream(socket);
> socketWriter = new OpenWireBinaryWriter(networkStream);
> socketReader = new OpenWireBinaryReader(networkStream);
> This pattern is explicitly allowed in Java and Win32 applications, but .Net is a little vague on the subject. The Microsoft documentation [1] suggests use of the asynchronous read/write calls for multithreaded applications, but that would significantly complicate the C# client which relies on blocking stream behaviour. On the same doc page
> Microsoft does specifiy that the Socket class is thread-safe, which I take to mean that concurrent read/write calls are permitted, however the same is not true of NetworkStream [2].
> Perhaps it would be reasonable to expect NetworkStream to have no internal state other than the Socket it contains, but apparently this is not the case because I've identified that it is a corrupt NetworkStream which is causing my application to crash under load. After some experimentation and a fair amount of load testing, I think the solution is a simple change to the TcpTransport.start() method to use a separate NetworkStream for input and output operations. e.g. :-
> NetworkStream inputNetworkStream = new NetworkStream(socket);
> NetworkStream outputNetworkStream = new NetworkStream(socket);
> socketWriter = new OpenWireBinaryWriter(inputNetworkStream);
> socketReader = new OpenWireBinaryReader(outputNetworkStream);
> This works for me and my test client has now been running for 16 hours without crashing (before it would rarely last longer than 10 minutes).
> Regards
> Rob Lugt
> [1] http://msdn2.microsoft.com/en-us/library/system.net.sockets.socket.aspx
> [2] http://msdn2.microsoft.com/en-us/library/system.net.sockets.networkstream.aspx
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (AMQ-988) Thread synchronization error in
TcpTransport
Posted by "Rob Lugt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-988?page=all ]
Rob Lugt updated AMQ-988:
-------------------------
Attachment: amq988-patch.txt
Please apply the attached patch, which fixes a bug in OpenWireBinaryReader.ReadString.
This bug involves an incorrect call of the Read(byte[] buffer, int index, int count) method. Teh function requires that exactly the specified number of bytes are read from the input stream, but the way it is currently written it will read any amount up to the specified number of bytes (depending on what data is available in the socket input buffer).
The result of this bug is a corrupt message when the required data is not in the socket input buffer (which, for me, happens as soon as the message rate exceeds 100 msgs per second). The read-loop thread terminates and no further messages are received.
This is the true underlying cause of this bug. The earlier analysis on threading appears tro be superfluous (but it's probably worth keeping the dual NetworkStreams as this class is not advertised as threrad-safe).
Regards
Rob Lugt
> Thread synchronization error in TcpTransport
> --------------------------------------------
>
> Key: AMQ-988
> URL: https://issues.apache.org/activemq/browse/AMQ-988
> Project: ActiveMQ
> Issue Type: Bug
> Components: NMS (C# client)
> Affects Versions: 4.0.2
> Environment: Windows
> Reporter: Rob Lugt
> Assigned To: Rob Lugt
> Fix For: 4.1
>
> Attachments: amq988-patch.txt
>
>
> I have a problem where my C# client application crashes when placed under load. It's taken a while to get to the bottom of it, but I believe I have identified the problem and luckily there's a simple solution.
> The AMQ .Net client uses a common pattern where a full-duplex TCP/IP connection is established with the broker, and the client uses two threads to concurrently read and write data to/from the underlying socket - one thread reading from a Reader object and the other writing to a Writer object.
> The TcpTransport.Start() method contains the following:-
> NetworkStream networkStream = new NetworkStream(socket);
> socketWriter = new OpenWireBinaryWriter(networkStream);
> socketReader = new OpenWireBinaryReader(networkStream);
> This pattern is explicitly allowed in Java and Win32 applications, but .Net is a little vague on the subject. The Microsoft documentation [1] suggests use of the asynchronous read/write calls for multithreaded applications, but that would significantly complicate the C# client which relies on blocking stream behaviour. On the same doc page
> Microsoft does specifiy that the Socket class is thread-safe, which I take to mean that concurrent read/write calls are permitted, however the same is not true of NetworkStream [2].
> Perhaps it would be reasonable to expect NetworkStream to have no internal state other than the Socket it contains, but apparently this is not the case because I've identified that it is a corrupt NetworkStream which is causing my application to crash under load. After some experimentation and a fair amount of load testing, I think the solution is a simple change to the TcpTransport.start() method to use a separate NetworkStream for input and output operations. e.g. :-
> NetworkStream inputNetworkStream = new NetworkStream(socket);
> NetworkStream outputNetworkStream = new NetworkStream(socket);
> socketWriter = new OpenWireBinaryWriter(inputNetworkStream);
> socketReader = new OpenWireBinaryReader(outputNetworkStream);
> This works for me and my test client has now been running for 16 hours without crashing (before it would rarely last longer than 10 minutes).
> Regards
> Rob Lugt
> [1] http://msdn2.microsoft.com/en-us/library/system.net.sockets.socket.aspx
> [2] http://msdn2.microsoft.com/en-us/library/system.net.sockets.networkstream.aspx
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (AMQ-988) Thread synchronization error in
TcpTransport
Posted by "james strachan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-988?page=all ]
james strachan resolved AMQ-988.
--------------------------------
Fix Version/s: 4.1
Resolution: Fixed
Great catch Rob! Patch applied with huge thanks
> Thread synchronization error in TcpTransport
> --------------------------------------------
>
> Key: AMQ-988
> URL: https://issues.apache.org/activemq/browse/AMQ-988
> Project: ActiveMQ
> Issue Type: Bug
> Components: NMS (C# client)
> Affects Versions: 4.0.2
> Environment: Windows
> Reporter: Rob Lugt
> Assigned To: james strachan
> Fix For: 4.1
>
>
> I have a problem where my C# client application crashes when placed under load. It's taken a while to get to the bottom of it, but I believe I have identified the problem and luckily there's a simple solution.
> The AMQ .Net client uses a common pattern where a full-duplex TCP/IP connection is established with the broker, and the client uses two threads to concurrently read and write data to/from the underlying socket - one thread reading from a Reader object and the other writing to a Writer object.
> The TcpTransport.Start() method contains the following:-
> NetworkStream networkStream = new NetworkStream(socket);
> socketWriter = new OpenWireBinaryWriter(networkStream);
> socketReader = new OpenWireBinaryReader(networkStream);
> This pattern is explicitly allowed in Java and Win32 applications, but .Net is a little vague on the subject. The Microsoft documentation [1] suggests use of the asynchronous read/write calls for multithreaded applications, but that would significantly complicate the C# client which relies on blocking stream behaviour. On the same doc page
> Microsoft does specifiy that the Socket class is thread-safe, which I take to mean that concurrent read/write calls are permitted, however the same is not true of NetworkStream [2].
> Perhaps it would be reasonable to expect NetworkStream to have no internal state other than the Socket it contains, but apparently this is not the case because I've identified that it is a corrupt NetworkStream which is causing my application to crash under load. After some experimentation and a fair amount of load testing, I think the solution is a simple change to the TcpTransport.start() method to use a separate NetworkStream for input and output operations. e.g. :-
> NetworkStream inputNetworkStream = new NetworkStream(socket);
> NetworkStream outputNetworkStream = new NetworkStream(socket);
> socketWriter = new OpenWireBinaryWriter(inputNetworkStream);
> socketReader = new OpenWireBinaryReader(outputNetworkStream);
> This works for me and my test client has now been running for 16 hours without crashing (before it would rarely last longer than 10 minutes).
> Regards
> Rob Lugt
> [1] http://msdn2.microsoft.com/en-us/library/system.net.sockets.socket.aspx
> [2] http://msdn2.microsoft.com/en-us/library/system.net.sockets.networkstream.aspx
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Assigned: (AMQ-988) Thread synchronization error in
TcpTransport
Posted by "Rob Lugt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-988?page=all ]
Rob Lugt reassigned AMQ-988:
----------------------------
Assignee: james strachan (was: Rob Lugt)
Hi James. Please review and apply supplied patch.
> Thread synchronization error in TcpTransport
> --------------------------------------------
>
> Key: AMQ-988
> URL: https://issues.apache.org/activemq/browse/AMQ-988
> Project: ActiveMQ
> Issue Type: Bug
> Components: NMS (C# client)
> Affects Versions: 4.0.2
> Environment: Windows
> Reporter: Rob Lugt
> Assigned To: james strachan
> Fix For: 4.1
>
> Attachments: amq988-patch.txt
>
>
> I have a problem where my C# client application crashes when placed under load. It's taken a while to get to the bottom of it, but I believe I have identified the problem and luckily there's a simple solution.
> The AMQ .Net client uses a common pattern where a full-duplex TCP/IP connection is established with the broker, and the client uses two threads to concurrently read and write data to/from the underlying socket - one thread reading from a Reader object and the other writing to a Writer object.
> The TcpTransport.Start() method contains the following:-
> NetworkStream networkStream = new NetworkStream(socket);
> socketWriter = new OpenWireBinaryWriter(networkStream);
> socketReader = new OpenWireBinaryReader(networkStream);
> This pattern is explicitly allowed in Java and Win32 applications, but .Net is a little vague on the subject. The Microsoft documentation [1] suggests use of the asynchronous read/write calls for multithreaded applications, but that would significantly complicate the C# client which relies on blocking stream behaviour. On the same doc page
> Microsoft does specifiy that the Socket class is thread-safe, which I take to mean that concurrent read/write calls are permitted, however the same is not true of NetworkStream [2].
> Perhaps it would be reasonable to expect NetworkStream to have no internal state other than the Socket it contains, but apparently this is not the case because I've identified that it is a corrupt NetworkStream which is causing my application to crash under load. After some experimentation and a fair amount of load testing, I think the solution is a simple change to the TcpTransport.start() method to use a separate NetworkStream for input and output operations. e.g. :-
> NetworkStream inputNetworkStream = new NetworkStream(socket);
> NetworkStream outputNetworkStream = new NetworkStream(socket);
> socketWriter = new OpenWireBinaryWriter(inputNetworkStream);
> socketReader = new OpenWireBinaryReader(outputNetworkStream);
> This works for me and my test client has now been running for 16 hours without crashing (before it would rarely last longer than 10 minutes).
> Regards
> Rob Lugt
> [1] http://msdn2.microsoft.com/en-us/library/system.net.sockets.socket.aspx
> [2] http://msdn2.microsoft.com/en-us/library/system.net.sockets.networkstream.aspx
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Assigned: (AMQ-988) Thread synchronization error in
TcpTransport
Posted by "Rob Lugt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-988?page=all ]
Rob Lugt reassigned AMQ-988:
----------------------------
Assignee: james strachan
> Thread synchronization error in TcpTransport
> --------------------------------------------
>
> Key: AMQ-988
> URL: https://issues.apache.org/activemq/browse/AMQ-988
> Project: ActiveMQ
> Issue Type: Bug
> Components: NMS (C# client)
> Affects Versions: 4.0.2
> Environment: Windows
> Reporter: Rob Lugt
> Assigned To: james strachan
>
> I have a problem where my C# client application crashes when placed under load. It's taken a while to get to the bottom of it, but I believe I have identified the problem and luckily there's a simple solution.
> The AMQ .Net client uses a common pattern where a full-duplex TCP/IP connection is established with the broker, and the client uses two threads to concurrently read and write data to/from the underlying socket - one thread reading from a Reader object and the other writing to a Writer object.
> The TcpTransport.Start() method contains the following:-
> NetworkStream networkStream = new NetworkStream(socket);
> socketWriter = new OpenWireBinaryWriter(networkStream);
> socketReader = new OpenWireBinaryReader(networkStream);
> This pattern is explicitly allowed in Java and Win32 applications, but .Net is a little vague on the subject. The Microsoft documentation [1] suggests use of the asynchronous read/write calls for multithreaded applications, but that would significantly complicate the C# client which relies on blocking stream behaviour. On the same doc page
> Microsoft does specifiy that the Socket class is thread-safe, which I take to mean that concurrent read/write calls are permitted, however the same is not true of NetworkStream [2].
> Perhaps it would be reasonable to expect NetworkStream to have no internal state other than the Socket it contains, but apparently this is not the case because I've identified that it is a corrupt NetworkStream which is causing my application to crash under load. After some experimentation and a fair amount of load testing, I think the solution is a simple change to the TcpTransport.start() method to use a separate NetworkStream for input and output operations. e.g. :-
> NetworkStream inputNetworkStream = new NetworkStream(socket);
> NetworkStream outputNetworkStream = new NetworkStream(socket);
> socketWriter = new OpenWireBinaryWriter(inputNetworkStream);
> socketReader = new OpenWireBinaryReader(outputNetworkStream);
> This works for me and my test client has now been running for 16 hours without crashing (before it would rarely last longer than 10 minutes).
> Regards
> Rob Lugt
> [1] http://msdn2.microsoft.com/en-us/library/system.net.sockets.socket.aspx
> [2] http://msdn2.microsoft.com/en-us/library/system.net.sockets.networkstream.aspx
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira