You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by "Patrick Hunt (JIRA)" <ji...@apache.org> on 2009/10/26 21:46:59 UTC
[jira] Created: (ZOOKEEPER-562) c client can flood server with
pings if tcp send queue filled
c client can flood server with pings if tcp send queue filled
-------------------------------------------------------------
Key: ZOOKEEPER-562
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-562
Project: Zookeeper
Issue Type: Bug
Components: c client
Affects Versions: 3.2.1
Reporter: Patrick Hunt
Assignee: Mahadev konar
Priority: Blocker
Fix For: 3.2.2, 3.3.0
The c client can flood the server with pings if the tcp queue is filled.
Say the cluster is overloaded and shuts down the recv processing
a c client can send a ping, but since last_send is only updated on successful pushing of data into the
socket, if flush_send_queue fails to send any data (send_buffer returns 0) then last_send is not updated
and zookeeper_interest will again send a ping the next time it is woken - which could be 0 if recv_to is close
to 0, easily could happen if server is not sending data to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-562) c client can flood server with
pings if tcp send queue filled
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771607#action_12771607 ]
Mahadev konar commented on ZOOKEEPER-562:
-----------------------------------------
+1 this looks good.
> c client can flood server with pings if tcp send queue filled
> -------------------------------------------------------------
>
> Key: ZOOKEEPER-562
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-562
> Project: Zookeeper
> Issue Type: Bug
> Components: c client
> Affects Versions: 3.2.1
> Reporter: Patrick Hunt
> Assignee: Benjamin Reed
> Priority: Blocker
> Fix For: 3.2.2, 3.3.0
>
> Attachments: ZOOKEEPER-562.patch
>
>
> The c client can flood the server with pings if the tcp queue is filled.
> Say the cluster is overloaded and shuts down the recv processing
> a c client can send a ping, but since last_send is only updated on successful pushing of data into the
> socket, if flush_send_queue fails to send any data (send_buffer returns 0) then last_send is not updated
> and zookeeper_interest will again send a ping the next time it is woken - which could be 0 if recv_to is close
> to 0, easily could happen if server is not sending data to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-562) c client can flood server with
pings if tcp send queue filled
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770398#action_12770398 ]
Hadoop QA commented on ZOOKEEPER-562:
-------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12423282/ZOOKEEPER-562.patch
against trunk revision 828216.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/40/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/40/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/40/console
This message is automatically generated.
> c client can flood server with pings if tcp send queue filled
> -------------------------------------------------------------
>
> Key: ZOOKEEPER-562
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-562
> Project: Zookeeper
> Issue Type: Bug
> Components: c client
> Affects Versions: 3.2.1
> Reporter: Patrick Hunt
> Assignee: Benjamin Reed
> Priority: Blocker
> Fix For: 3.2.2, 3.3.0
>
> Attachments: ZOOKEEPER-562.patch
>
>
> The c client can flood the server with pings if the tcp queue is filled.
> Say the cluster is overloaded and shuts down the recv processing
> a c client can send a ping, but since last_send is only updated on successful pushing of data into the
> socket, if flush_send_queue fails to send any data (send_buffer returns 0) then last_send is not updated
> and zookeeper_interest will again send a ping the next time it is woken - which could be 0 if recv_to is close
> to 0, easily could happen if server is not sending data to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-562) c client can flood server with
pings if tcp send queue filled
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benjamin Reed updated ZOOKEEPER-562:
------------------------------------
Status: Patch Available (was: Open)
> c client can flood server with pings if tcp send queue filled
> -------------------------------------------------------------
>
> Key: ZOOKEEPER-562
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-562
> Project: Zookeeper
> Issue Type: Bug
> Components: c client
> Affects Versions: 3.2.1
> Reporter: Patrick Hunt
> Assignee: Benjamin Reed
> Priority: Blocker
> Fix For: 3.2.2, 3.3.0
>
> Attachments: ZOOKEEPER-562.patch
>
>
> The c client can flood the server with pings if the tcp queue is filled.
> Say the cluster is overloaded and shuts down the recv processing
> a c client can send a ping, but since last_send is only updated on successful pushing of data into the
> socket, if flush_send_queue fails to send any data (send_buffer returns 0) then last_send is not updated
> and zookeeper_interest will again send a ping the next time it is woken - which could be 0 if recv_to is close
> to 0, easily could happen if server is not sending data to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-562) c client can flood server with
pings if tcp send queue filled
Posted by "Benjamin Reed (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benjamin Reed updated ZOOKEEPER-562:
------------------------------------
Attachment: ZOOKEEPER-562.patch
this patch fixes the problem by only sending a ping if there isn't something already queued. the test checks for clients sending gratuitous pings.
> c client can flood server with pings if tcp send queue filled
> -------------------------------------------------------------
>
> Key: ZOOKEEPER-562
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-562
> Project: Zookeeper
> Issue Type: Bug
> Components: c client
> Affects Versions: 3.2.1
> Reporter: Patrick Hunt
> Assignee: Benjamin Reed
> Priority: Blocker
> Fix For: 3.2.2, 3.3.0
>
> Attachments: ZOOKEEPER-562.patch
>
>
> The c client can flood the server with pings if the tcp queue is filled.
> Say the cluster is overloaded and shuts down the recv processing
> a c client can send a ping, but since last_send is only updated on successful pushing of data into the
> socket, if flush_send_queue fails to send any data (send_buffer returns 0) then last_send is not updated
> and zookeeper_interest will again send a ping the next time it is woken - which could be 0 if recv_to is close
> to 0, easily could happen if server is not sending data to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-562) c client can flood server with
pings if tcp send queue filled
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated ZOOKEEPER-562:
-----------------------------------
Release Note: Basically the problem here is that the client gets confused, it tries to send a ping to the server but since the tcp queue is full it's unable to do so. The logic responsible for sending occasional pings based on the timeout gets confused by this, and ends up flooding the server with pings. Eventually this clears up, however it can result in increased load on the server and instability for the effected client.
> c client can flood server with pings if tcp send queue filled
> -------------------------------------------------------------
>
> Key: ZOOKEEPER-562
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-562
> Project: Zookeeper
> Issue Type: Bug
> Components: c client
> Affects Versions: 3.2.1
> Reporter: Patrick Hunt
> Assignee: Benjamin Reed
> Priority: Blocker
> Fix For: 3.2.2, 3.3.0
>
> Attachments: ZOOKEEPER-562.patch
>
>
> The c client can flood the server with pings if the tcp queue is filled.
> Say the cluster is overloaded and shuts down the recv processing
> a c client can send a ping, but since last_send is only updated on successful pushing of data into the
> socket, if flush_send_queue fails to send any data (send_buffer returns 0) then last_send is not updated
> and zookeeper_interest will again send a ping the next time it is woken - which could be 0 if recv_to is close
> to 0, easily could happen if server is not sending data to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-562) c client can flood server with
pings if tcp send queue filled
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771910#action_12771910 ]
Hudson commented on ZOOKEEPER-562:
----------------------------------
Integrated in ZooKeeper-trunk #513 (See [http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/513/])
. c client can flood server with pings if tcp send queue filled. (ben reed via mahadev)
> c client can flood server with pings if tcp send queue filled
> -------------------------------------------------------------
>
> Key: ZOOKEEPER-562
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-562
> Project: Zookeeper
> Issue Type: Bug
> Components: c client
> Affects Versions: 3.2.1
> Reporter: Patrick Hunt
> Assignee: Benjamin Reed
> Priority: Blocker
> Fix For: 3.2.2, 3.3.0
>
> Attachments: ZOOKEEPER-562.patch
>
>
> The c client can flood the server with pings if the tcp queue is filled.
> Say the cluster is overloaded and shuts down the recv processing
> a c client can send a ping, but since last_send is only updated on successful pushing of data into the
> socket, if flush_send_queue fails to send any data (send_buffer returns 0) then last_send is not updated
> and zookeeper_interest will again send a ping the next time it is woken - which could be 0 if recv_to is close
> to 0, easily could happen if server is not sending data to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-562) c client can flood server with
pings if tcp send queue filled
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12770291#action_12770291 ]
Patrick Hunt commented on ZOOKEEPER-562:
----------------------------------------
send_ping is calling wake_io_thread itself, so this is a particularly bad situation (forces a tight loop)
solution is to update last_send as last_send_attempt when attempting to send, whether successful or not.
> c client can flood server with pings if tcp send queue filled
> -------------------------------------------------------------
>
> Key: ZOOKEEPER-562
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-562
> Project: Zookeeper
> Issue Type: Bug
> Components: c client
> Affects Versions: 3.2.1
> Reporter: Patrick Hunt
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 3.2.2, 3.3.0
>
>
> The c client can flood the server with pings if the tcp queue is filled.
> Say the cluster is overloaded and shuts down the recv processing
> a c client can send a ping, but since last_send is only updated on successful pushing of data into the
> socket, if flush_send_queue fails to send any data (send_buffer returns 0) then last_send is not updated
> and zookeeper_interest will again send a ping the next time it is woken - which could be 0 if recv_to is close
> to 0, easily could happen if server is not sending data to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-562) c client can flood server with
pings if tcp send queue filled
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated ZOOKEEPER-562:
------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this to 3.2 branch and trunk.
> c client can flood server with pings if tcp send queue filled
> -------------------------------------------------------------
>
> Key: ZOOKEEPER-562
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-562
> Project: Zookeeper
> Issue Type: Bug
> Components: c client
> Affects Versions: 3.2.1
> Reporter: Patrick Hunt
> Assignee: Benjamin Reed
> Priority: Blocker
> Fix For: 3.2.2, 3.3.0
>
> Attachments: ZOOKEEPER-562.patch
>
>
> The c client can flood the server with pings if the tcp queue is filled.
> Say the cluster is overloaded and shuts down the recv processing
> a c client can send a ping, but since last_send is only updated on successful pushing of data into the
> socket, if flush_send_queue fails to send any data (send_buffer returns 0) then last_send is not updated
> and zookeeper_interest will again send a ping the next time it is woken - which could be 0 if recv_to is close
> to 0, easily could happen if server is not sending data to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-562) c client can flood server with
pings if tcp send queue filled
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated ZOOKEEPER-562:
-----------------------------------
Fix Version/s: 3.1.2
> c client can flood server with pings if tcp send queue filled
> -------------------------------------------------------------
>
> Key: ZOOKEEPER-562
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-562
> Project: Zookeeper
> Issue Type: Bug
> Components: c client
> Affects Versions: 3.2.1
> Reporter: Patrick Hunt
> Assignee: Benjamin Reed
> Priority: Blocker
> Fix For: 3.2.2, 3.3.0, 3.1.2
>
> Attachments: ZOOKEEPER-562.patch
>
>
> The c client can flood the server with pings if the tcp queue is filled.
> Say the cluster is overloaded and shuts down the recv processing
> a c client can send a ping, but since last_send is only updated on successful pushing of data into the
> socket, if flush_send_queue fails to send any data (send_buffer returns 0) then last_send is not updated
> and zookeeper_interest will again send a ping the next time it is woken - which could be 0 if recv_to is close
> to 0, easily could happen if server is not sending data to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-562) c client can flood server with
pings if tcp send queue filled
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12771295#action_12771295 ]
Patrick Hunt commented on ZOOKEEPER-562:
----------------------------------------
+1, looks good to me - if we are waiting for a result we don't need to send a ping. great!
Mahadev, can you take a look at this and commit if no issues found? (both 3.2 branch and trunk)
> c client can flood server with pings if tcp send queue filled
> -------------------------------------------------------------
>
> Key: ZOOKEEPER-562
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-562
> Project: Zookeeper
> Issue Type: Bug
> Components: c client
> Affects Versions: 3.2.1
> Reporter: Patrick Hunt
> Assignee: Benjamin Reed
> Priority: Blocker
> Fix For: 3.2.2, 3.3.0
>
> Attachments: ZOOKEEPER-562.patch
>
>
> The c client can flood the server with pings if the tcp queue is filled.
> Say the cluster is overloaded and shuts down the recv processing
> a c client can send a ping, but since last_send is only updated on successful pushing of data into the
> socket, if flush_send_queue fails to send any data (send_buffer returns 0) then last_send is not updated
> and zookeeper_interest will again send a ping the next time it is woken - which could be 0 if recv_to is close
> to 0, easily could happen if server is not sending data to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (ZOOKEEPER-562) c client can flood server with
pings if tcp send queue filled
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt reassigned ZOOKEEPER-562:
--------------------------------------
Assignee: Benjamin Reed (was: Mahadev konar)
Assigning to Ben.
We should verify that something like this can't happen in the java either. From my looking
it seems not, but would be good to have addl verification.
> c client can flood server with pings if tcp send queue filled
> -------------------------------------------------------------
>
> Key: ZOOKEEPER-562
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-562
> Project: Zookeeper
> Issue Type: Bug
> Components: c client
> Affects Versions: 3.2.1
> Reporter: Patrick Hunt
> Assignee: Benjamin Reed
> Priority: Blocker
> Fix For: 3.2.2, 3.3.0
>
>
> The c client can flood the server with pings if the tcp queue is filled.
> Say the cluster is overloaded and shuts down the recv processing
> a c client can send a ping, but since last_send is only updated on successful pushing of data into the
> socket, if flush_send_queue fails to send any data (send_buffer returns 0) then last_send is not updated
> and zookeeper_interest will again send a ping the next time it is woken - which could be 0 if recv_to is close
> to 0, easily could happen if server is not sending data to the client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.