You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jun Rao (JIRA)" <ji...@apache.org> on 2009/06/08 06:18:07 UTC
[jira] Created: (CASSANDRA-220) TCP writes get stuck
TCP writes get stuck
--------------------
Key: CASSANDRA-220
URL: https://issues.apache.org/jira/browse/CASSANDRA-220
Project: Cassandra
Issue Type: Bug
Affects Versions: trunk
Reporter: Jun Rao
Assignee: Jun Rao
Fix For: trunk
In our test cluster, I observed that on some nodes, TCP writes get accumulated in TcpConnection.pendingWrites. However, the selector never gets a ready to write event. As a result, all writes get stuck. This is because write(message) and doPendingWrites() are not fully synchronized. This following situation can happen:
1. write(message) adds stuff to pendingWrites.
2. a ready to write event happens; in doPendingWrites() buffered requests are written to socket
3. another write request happens, in write(message), the test for pendingWrites.isEmpty() is false
4. doPendingWrites() finishes writing all buffered request to socket
5. in write(message), the new request is added to pendingWrites
Now, ready to write events will never happen again and all write requests get stuck in pendingWrites.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-220) TCP writes get stuck
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated CASSANDRA-220:
------------------------------
Attachment: 220-5.patch
Attach patch-5 that fixes similar problems in HttpConnection and UdpConnection.
> TCP writes get stuck
> --------------------
>
> Key: CASSANDRA-220
> URL: https://issues.apache.org/jira/browse/CASSANDRA-220
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jun Rao
> Assignee: Jun Rao
> Attachments: 220-2.patch, 220-3.patch, 220-4.patch, 220-5.patch, issue220.patchv1
>
>
> In our test cluster, I observed that on some nodes, TCP writes get accumulated in TcpConnection.pendingWrites. However, the selector never gets a ready to write event. As a result, all writes get stuck. This is because write(message) and doPendingWrites() are not fully synchronized. This following situation can happen:
> 1. write(message) adds stuff to pendingWrites.
> 2. a ready to write event happens; in doPendingWrites() buffered requests are written to socket
> 3. another write request happens, in write(message), the test for pendingWrites.isEmpty() is false
> 4. doPendingWrites() finishes writing all buffered request to socket
> 5. in write(message), the new request is added to pendingWrites
> Now, ready to write events will never happen again and all write requests get stuck in pendingWrites.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-220) TCP writes get stuck
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-220:
-------------------------------------
Attachment: 220-2.patch
Committed to 0.3. Will merge to trunk.
Here is another patch to clean up TcpConnection a little. Look okay?
> TCP writes get stuck
> --------------------
>
> Key: CASSANDRA-220
> URL: https://issues.apache.org/jira/browse/CASSANDRA-220
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Reporter: Jun Rao
> Assignee: Jun Rao
> Fix For: trunk
>
> Attachments: 220-2.patch, issue220.patchv1
>
>
> In our test cluster, I observed that on some nodes, TCP writes get accumulated in TcpConnection.pendingWrites. However, the selector never gets a ready to write event. As a result, all writes get stuck. This is because write(message) and doPendingWrites() are not fully synchronized. This following situation can happen:
> 1. write(message) adds stuff to pendingWrites.
> 2. a ready to write event happens; in doPendingWrites() buffered requests are written to socket
> 3. another write request happens, in write(message), the test for pendingWrites.isEmpty() is false
> 4. doPendingWrites() finishes writing all buffered request to socket
> 5. in write(message), the new request is added to pendingWrites
> Now, ready to write events will never happen again and all write requests get stuck in pendingWrites.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-220) TCP writes get stuck
Posted by "Michael Greene (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Greene updated CASSANDRA-220:
-------------------------------------
Component/s: Core
> TCP writes get stuck
> --------------------
>
> Key: CASSANDRA-220
> URL: https://issues.apache.org/jira/browse/CASSANDRA-220
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Jun Rao
> Assignee: Jun Rao
> Fix For: 0.4
>
> Attachments: 220-2.patch, 220-3.patch, 220-4.patch, 220-5.patch, issue220.patchv1
>
>
> In our test cluster, I observed that on some nodes, TCP writes get accumulated in TcpConnection.pendingWrites. However, the selector never gets a ready to write event. As a result, all writes get stuck. This is because write(message) and doPendingWrites() are not fully synchronized. This following situation can happen:
> 1. write(message) adds stuff to pendingWrites.
> 2. a ready to write event happens; in doPendingWrites() buffered requests are written to socket
> 3. another write request happens, in write(message), the test for pendingWrites.isEmpty() is false
> 4. doPendingWrites() finishes writing all buffered request to socket
> 5. in write(message), the new request is added to pendingWrites
> Now, ready to write events will never happen again and all write requests get stuck in pendingWrites.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-220) TCP writes get stuck
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719048#action_12719048 ]
Jonathan Ellis commented on CASSANDRA-220:
------------------------------------------
I know nio is sexier, but maybe it would be better to rip it out and just go with thread-per-socket.
http://java.dzone.com/articles/avoid-nio-get-better-throughpu (read the linked slides)
> TCP writes get stuck
> --------------------
>
> Key: CASSANDRA-220
> URL: https://issues.apache.org/jira/browse/CASSANDRA-220
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Reporter: Jun Rao
> Assignee: Jun Rao
> Fix For: trunk
>
> Attachments: 220-2.patch, 220-3.patch, issue220.patchv1
>
>
> In our test cluster, I observed that on some nodes, TCP writes get accumulated in TcpConnection.pendingWrites. However, the selector never gets a ready to write event. As a result, all writes get stuck. This is because write(message) and doPendingWrites() are not fully synchronized. This following situation can happen:
> 1. write(message) adds stuff to pendingWrites.
> 2. a ready to write event happens; in doPendingWrites() buffered requests are written to socket
> 3. another write request happens, in write(message), the test for pendingWrites.isEmpty() is false
> 4. doPendingWrites() finishes writing all buffered request to socket
> 5. in write(message), the new request is added to pendingWrites
> Now, ready to write events will never happen again and all write requests get stuck in pendingWrites.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-220) TCP writes get stuck
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717292#action_12717292 ]
Jun Rao commented on CASSANDRA-220:
-----------------------------------
The new patch looks fine to me.
> TCP writes get stuck
> --------------------
>
> Key: CASSANDRA-220
> URL: https://issues.apache.org/jira/browse/CASSANDRA-220
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Reporter: Jun Rao
> Assignee: Jun Rao
> Fix For: trunk
>
> Attachments: 220-2.patch, issue220.patchv1
>
>
> In our test cluster, I observed that on some nodes, TCP writes get accumulated in TcpConnection.pendingWrites. However, the selector never gets a ready to write event. As a result, all writes get stuck. This is because write(message) and doPendingWrites() are not fully synchronized. This following situation can happen:
> 1. write(message) adds stuff to pendingWrites.
> 2. a ready to write event happens; in doPendingWrites() buffered requests are written to socket
> 3. another write request happens, in write(message), the test for pendingWrites.isEmpty() is false
> 4. doPendingWrites() finishes writing all buffered request to socket
> 5. in write(message), the new request is added to pendingWrites
> Now, ready to write events will never happen again and all write requests get stuck in pendingWrites.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (CASSANDRA-220) TCP writes get stuck
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao reopened CASSANDRA-220:
-------------------------------
> TCP writes get stuck
> --------------------
>
> Key: CASSANDRA-220
> URL: https://issues.apache.org/jira/browse/CASSANDRA-220
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Reporter: Jun Rao
> Assignee: Jun Rao
> Fix For: trunk
>
> Attachments: 220-2.patch, 220-3.patch, issue220.patchv1
>
>
> In our test cluster, I observed that on some nodes, TCP writes get accumulated in TcpConnection.pendingWrites. However, the selector never gets a ready to write event. As a result, all writes get stuck. This is because write(message) and doPendingWrites() are not fully synchronized. This following situation can happen:
> 1. write(message) adds stuff to pendingWrites.
> 2. a ready to write event happens; in doPendingWrites() buffered requests are written to socket
> 3. another write request happens, in write(message), the test for pendingWrites.isEmpty() is false
> 4. doPendingWrites() finishes writing all buffered request to socket
> 5. in write(message), the new request is added to pendingWrites
> Now, ready to write events will never happen again and all write requests get stuck in pendingWrites.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-220) TCP writes get stuck
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated CASSANDRA-220:
------------------------------
Attachment: 220-4.patch
It took me a while to dig this out. Under heavy load and large column values, we still saw lockups in tcp connection. Here is the problem. The following code that sets the interest ops seems innocent, but it's the source of the problem. The reason is that this operation is not atomic. Another thread could sneak in between the reading of the ops and the setting of it. As a result, some wrong bits could be set.
key_.interestOps(key_.interestOps() | SelectionKey.OP_READ)
This is a sequence that demonstrates how we can lose the OP_READ bit forever and thus jam the read channel:
1. Thread 1: we want to write a message and in write(Message) we are about to turn on OP_WRITE because the message can't be written in one shot.
2. Thread 2: a read comes in and in read(SelectionKey), we turn off OP_READ and submit the read request to ReadWorkItem in Thread 3.
3. Thread 1: read interestOps and see OP_READ as off.
4. Thread 3: finished processing the read request and turn OP_READ on
5. Thread 1: resumes and turn on OP_WRITE. However, by doing that, we also turned off OP_READ. The read channel is thus blocked forever after this.
Patch-4 makes changing interestOps atomic by synchronizing on the selection key.
Patch-4 also includes the earlier fix for the synchronization problem in connect. I left the fix as it is instead of making connect() a synchronized method. This way, it makes clear which part of the code in connect really requires synchronization.
> TCP writes get stuck
> --------------------
>
> Key: CASSANDRA-220
> URL: https://issues.apache.org/jira/browse/CASSANDRA-220
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jun Rao
> Assignee: Jun Rao
> Attachments: 220-2.patch, 220-3.patch, 220-4.patch, issue220.patchv1
>
>
> In our test cluster, I observed that on some nodes, TCP writes get accumulated in TcpConnection.pendingWrites. However, the selector never gets a ready to write event. As a result, all writes get stuck. This is because write(message) and doPendingWrites() are not fully synchronized. This following situation can happen:
> 1. write(message) adds stuff to pendingWrites.
> 2. a ready to write event happens; in doPendingWrites() buffered requests are written to socket
> 3. another write request happens, in write(message), the test for pendingWrites.isEmpty() is false
> 4. doPendingWrites() finishes writing all buffered request to socket
> 5. in write(message), the new request is added to pendingWrites
> Now, ready to write events will never happen again and all write requests get stuck in pendingWrites.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-220) TCP writes get stuck
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12723177#action_12723177 ]
Jonathan Ellis commented on CASSANDRA-220:
------------------------------------------
looks good.
can you patch the other uses of interestOps in UDPConnection and HttpConnection too?
> TCP writes get stuck
> --------------------
>
> Key: CASSANDRA-220
> URL: https://issues.apache.org/jira/browse/CASSANDRA-220
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jun Rao
> Assignee: Jun Rao
> Attachments: 220-2.patch, 220-3.patch, 220-4.patch, issue220.patchv1
>
>
> In our test cluster, I observed that on some nodes, TCP writes get accumulated in TcpConnection.pendingWrites. However, the selector never gets a ready to write event. As a result, all writes get stuck. This is because write(message) and doPendingWrites() are not fully synchronized. This following situation can happen:
> 1. write(message) adds stuff to pendingWrites.
> 2. a ready to write event happens; in doPendingWrites() buffered requests are written to socket
> 3. another write request happens, in write(message), the test for pendingWrites.isEmpty() is false
> 4. doPendingWrites() finishes writing all buffered request to socket
> 5. in write(message), the new request is added to pendingWrites
> Now, ready to write events will never happen again and all write requests get stuck in pendingWrites.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-220) TCP writes get stuck
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated CASSANDRA-220:
------------------------------
Attachment: 220-3.patch
Haven't beaten this one to death yet. There is another potential synchronization problem with respect to TcpConnection.connect. The following could happen:
1. Thread 1: TcpConnection.write(message) checks the channel and finds it not connected yet.
2. Thread 2: Sockect is connected and triggers TcpConnection.connect; pendingWrites is tested to be empty and "interested in write" bit is not set
3. Thread 1: adds message to pendingWrites
Now, pendingWrites is not empty and the "read for write" event is never going to happen since we didn't register an interest for it. All subsequent writes get stuck afterward.
Attach a patch in 220-3. It synchronizes in TcpConnection.connect too. Should patch 0.3 branch too.
> TCP writes get stuck
> --------------------
>
> Key: CASSANDRA-220
> URL: https://issues.apache.org/jira/browse/CASSANDRA-220
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Reporter: Jun Rao
> Assignee: Jun Rao
> Fix For: trunk
>
> Attachments: 220-2.patch, 220-3.patch, issue220.patchv1
>
>
> In our test cluster, I observed that on some nodes, TCP writes get accumulated in TcpConnection.pendingWrites. However, the selector never gets a ready to write event. As a result, all writes get stuck. This is because write(message) and doPendingWrites() are not fully synchronized. This following situation can happen:
> 1. write(message) adds stuff to pendingWrites.
> 2. a ready to write event happens; in doPendingWrites() buffered requests are written to socket
> 3. another write request happens, in write(message), the test for pendingWrites.isEmpty() is false
> 4. doPendingWrites() finishes writing all buffered request to socket
> 5. in write(message), the new request is added to pendingWrites
> Now, ready to write events will never happen again and all write requests get stuck in pendingWrites.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-220) TCP writes get stuck
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12720818#action_12720818 ]
Jonathan Ellis commented on CASSANDRA-220:
------------------------------------------
on -3, my only suggestion is, maybe we should just synchronize the whole method for simplicity since it is only going to be called once.
what do you think?
> TCP writes get stuck
> --------------------
>
> Key: CASSANDRA-220
> URL: https://issues.apache.org/jira/browse/CASSANDRA-220
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Reporter: Jun Rao
> Assignee: Jun Rao
> Fix For: trunk
>
> Attachments: 220-2.patch, 220-3.patch, issue220.patchv1
>
>
> In our test cluster, I observed that on some nodes, TCP writes get accumulated in TcpConnection.pendingWrites. However, the selector never gets a ready to write event. As a result, all writes get stuck. This is because write(message) and doPendingWrites() are not fully synchronized. This following situation can happen:
> 1. write(message) adds stuff to pendingWrites.
> 2. a ready to write event happens; in doPendingWrites() buffered requests are written to socket
> 3. another write request happens, in write(message), the test for pendingWrites.isEmpty() is false
> 4. doPendingWrites() finishes writing all buffered request to socket
> 5. in write(message), the new request is added to pendingWrites
> Now, ready to write events will never happen again and all write requests get stuck in pendingWrites.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-220) TCP writes get stuck
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated CASSANDRA-220:
------------------------------
Attachment: issue220.patchv1
Attache a patch to trunk by expanding the synchronization scope in TcpConnection.doPendingWrites().
We need to patch to 0.3 branch too.
Also, we need to revisit the code in TcpConnection. Under heavy writes, is it better to keep accumulating writes until we run out of memory, or to put a cap on the write buffer and simply block and possibly timeout the writer.
> TCP writes get stuck
> --------------------
>
> Key: CASSANDRA-220
> URL: https://issues.apache.org/jira/browse/CASSANDRA-220
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Reporter: Jun Rao
> Assignee: Jun Rao
> Fix For: trunk
>
> Attachments: issue220.patchv1
>
>
> In our test cluster, I observed that on some nodes, TCP writes get accumulated in TcpConnection.pendingWrites. However, the selector never gets a ready to write event. As a result, all writes get stuck. This is because write(message) and doPendingWrites() are not fully synchronized. This following situation can happen:
> 1. write(message) adds stuff to pendingWrites.
> 2. a ready to write event happens; in doPendingWrites() buffered requests are written to socket
> 3. another write request happens, in write(message), the test for pendingWrites.isEmpty() is false
> 4. doPendingWrites() finishes writing all buffered request to socket
> 5. in write(message), the new request is added to pendingWrites
> Now, ready to write events will never happen again and all write requests get stuck in pendingWrites.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.