You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by bgedik <gi...@git.apache.org> on 2018/02/19 20:07:21 UTC
[GitHub] thrift pull request #1497: Remove workSocket call that is too early
GitHub user bgedik opened a pull request:
https://github.com/apache/thrift/pull/1497
Remove workSocket call that is too early
Make the work socket conditional
Revert back the changes
Only call workSocket() when there is pending data to read
Minor update to a comment
Only call workSocket() when there is pending data to read
Only call workSocket() when there is pending data to read
Fix the CMake build
Fix the CMake build
Fix the CMake build
Use the correct type for ioctlsocket call
Make non-blocking peek optional
Revert "Make non-blocking peek optional"
This reverts commit 501c4401ddb93cf9bf451dd289d4c3768c0a7b15.
Review fixes
Review fixes
Review fixes
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/bgedik/thrift bugfix/non-blocking-server-calls-workSocket-too-early
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/thrift/pull/1497.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1497
----
commit 6665029561d40d5b89b7ee9cb2487f6c905b904a
Author: Bugra Gedik <bg...@...>
Date: 2018-01-21T17:43:49Z
remove workSocket call that is too early
Make the work socket conditional
Revert back the changes
Only call workSocket() when there is pending data to read
Minor update to a comment
Only call workSocket() when there is pending data to read
Only call workSocket() when there is pending data to read
Fix the CMake build
Fix the CMake build
Fix the CMake build
Use the correct type for ioctlsocket call
Make non-blocking peek optional
Revert "Make non-blocking peek optional"
This reverts commit 501c4401ddb93cf9bf451dd289d4c3768c0a7b15.
Review fixes
Review fixes
Review fixes
----
---
[GitHub] thrift issue #1497: Do not call workSocket() in TNonblockigServer without en...
Posted by bgedik <gi...@git.apache.org>.
Github user bgedik commented on the issue:
https://github.com/apache/thrift/pull/1497
@jeking3 The ``client_pool_tests`` failed in 2 of the bots within Travis-CI. I don't think it is related to my change. Are these known to fail every now and then?
```
thrift.transport.base.TTransportException@src/thrift/transport/socket.d(255): Failed to connect to 127.0.0.1:9090.
----------------
??:? void thrift.transport.socket.TSocket.open() [0x29e331ca]
??:? int thrift.codegen.client_pool.TClientPool!(client_pool_test.ExTestService).TClientPool.executeOnPool!(int).executeOnPool(scope int delegate(thrift.codegen.client.TClientBase!(client_pool_test.ExTestService).TClientBase)) [0x29e204ca]
??:? int thrift.codegen.client_pool.TClientPool!(client_pool_test.ExTestService).TClientPool.getPort() [0x29e08916]
??:? void client_pool_test.syncClientPoolTest(const(ushort)[], client_pool_test.ExTestHandler[]) [0x29df090a]
??:? _Dmain [0x29df077b]
thrift.base.TCompoundOperationException@src/thrift/transport/socket.d(256): All addresses tried failed (127.0.0.1:9090: "Unable to connect socket: Connection refused", 127.0.0.1:9090: "Unable to connect socket: Connection refused", 127.0.0.1:9090: "Unable to connect socket: Connection refused").
FAIL: client_pool_test
```
---
[GitHub] thrift issue #1497: Do not call workSocket() in TNonblockigServer without en...
Posted by bgedik <gi...@git.apache.org>.
Github user bgedik commented on the issue:
https://github.com/apache/thrift/pull/1497
@jeking3 There is already a Jira item: https://issues.apache.org/jira/browse/THRIFT-4465
Btw, it makes sense that the fix resolves the framed transport failures. Because the bug was about the transition from the frame size reading to the frame content reading.
---
[GitHub] thrift issue #1497: Do not call workSocket() in TNonblockigServer without en...
Posted by jeking3 <gi...@git.apache.org>.
Github user jeking3 commented on the issue:
https://github.com/apache/thrift/pull/1497
Wow, haven't seen a green build in a while. Of course, none of the cross tests exercise nonblocking code. :( Would be nice to add this as another matrix option.
---
[GitHub] thrift issue #1497: Do not call workSocket() in TNonblockigServer without en...
Posted by jeking3 <gi...@git.apache.org>.
Github user jeking3 commented on the issue:
https://github.com/apache/thrift/pull/1497
Cool so that last test run was without the fix (I thought it had it, but it only had my fixes to TestServer.cpp), and adding your fix seems to have taken care of the header failures somehow:
```
root@1dcb0ce83732:/thrift/src# test/test.py --server cpp --client cpp --regex='.*framed.*-ip.*'
Apache Thrift - Integration Test Suite
Sun Mar 11 19:39:18 2018
===============================================================================
server-client: protocol: transport: result:
cpp-cpp compact framed-ip success
cpp-cpp multi-binary framed-ip-ssl success
cpp-cpp compact framed-ip-ssl success
cpp-cpp multi-binary framed-ip success
cpp-cpp multih framed-ip success
cpp-cpp multih framed-ip-ssl success
cpp-cpp multih-header framed-ip-ssl success
cpp-cpp multih-header framed-ip success
cpp-cpp multij-json framed-ip-ssl success
cpp-cpp multij-json framed-ip success
cpp-cpp json framed-ip-ssl success
cpp-cpp json framed-ip success
cpp-cpp multic framed-ip success
cpp-cpp multic framed-ip-ssl success
cpp-cpp multic-compact framed-ip-ssl success
cpp-cpp multic-compact framed-ip success
cpp-cpp multij-json framed-ip-ssl success
cpp-cpp multij-json framed-ip success
cpp-cpp multij framed-ip-ssl success
cpp-cpp multij framed-ip success
cpp-cpp header framed-ip-ssl success
cpp-cpp header framed-ip success
cpp-cpp multic-compact framed-ip-ssl success
cpp-cpp multic-compact framed-ip success
cpp-cpp multi-binary framed-ip-ssl success
cpp-cpp multi-binary framed-ip success
cpp-cpp multi framed-ip success
cpp-cpp multi framed-ip-ssl success
cpp-cpp multih-header framed-ip-ssl success
cpp-cpp multih-header framed-ip success
cpp-cpp binary framed-ip success
cpp-cpp binary framed-ip-ssl success
===============================================================================
No unexpected failures.
```
---
[GitHub] thrift issue #1497: Do not call workSocket() in TNonblockigServer without en...
Posted by ozymaxx <gi...@git.apache.org>.
Github user ozymaxx commented on the issue:
https://github.com/apache/thrift/pull/1497
I also encounter the same error, but it does not show up every time I run the tests on Travis CI.
---
[GitHub] thrift issue #1497: Do not call workSocket() in TNonblockigServer without en...
Posted by jeking3 <gi...@git.apache.org>.
Github user jeking3 commented on the issue:
https://github.com/apache/thrift/pull/1497
I'm not aware of those being common or known. I have a bit of work to do in order to stabilize the CI builds and get them back to green...
---
[GitHub] thrift issue #1497: Do not call workSocket() in TNonblockigServer without en...
Posted by bgedik <gi...@git.apache.org>.
Github user bgedik commented on the issue:
https://github.com/apache/thrift/pull/1497
@jeking3 I think all tests have passed.
---
[GitHub] thrift pull request #1497: Do not call workSocket() in TNonblockigServer wit...
Posted by jeking3 <gi...@git.apache.org>.
Github user jeking3 commented on a diff in the pull request:
https://github.com/apache/thrift/pull/1497#discussion_r172695014
--- Diff: lib/cpp/src/thrift/server/TNonblockingServer.cpp ---
@@ -472,6 +472,18 @@ void TNonblockingServer::TConnection::workSocket() {
}
// size known; now get the rest of the frame
transition();
+
+ // If the socket has more data than the frame header, continue to work on it. This is not strictly necessary for
+ // regular sockets, because if there is more data, libevent will fire the event handler registered for read
+ // readiness, which will in turn call workSocket(). However, some socket types (such as TSSLSocket) may have the
+ // data sitting in their internal buffers and from libevent's perspective, there is no further data available. In
+ // that case, not having this workSocket() call here would result in a hang as we will never get to work the socket,
+ // despite having more data.
+ if (tSocket_->hasPendingDataToRead())
--- End diff --
I am concerned this can cause an infinite loop if data is always available, since it calls itself.
---
[GitHub] thrift issue #1497: Do not call workSocket() in TNonblockigServer without en...
Posted by jeking3 <gi...@git.apache.org>.
Github user jeking3 commented on the issue:
https://github.com/apache/thrift/pull/1497
What's the Jira ticket ID for this work?
---
[GitHub] thrift pull request #1497: Do not call workSocket() in TNonblockigServer wit...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/thrift/pull/1497
---
[GitHub] thrift issue #1497: Do not call workSocket() in TNonblockigServer without en...
Posted by jeking3 <gi...@git.apache.org>.
Github user jeking3 commented on the issue:
https://github.com/apache/thrift/pull/1497
You need to create a Jira ticket in the THRIFT project (https://issues.apache.org/jira/projects/THRIFT/issues) that describes the bug and resolution so that I can merge it.
---
I was able to test these changes out by making some changes to the cpp TestServer for crosstest which I will check in today. I changed my local tests.json file to add "--server-type=nonblocking" to each cpp server and changed some code to make additional cases work:
```
root@8d560c4cc0fa:/thrift/src# test/test.py --server cpp --client cpp --regex='.*framed.*-ip.*'
Apache Thrift - Integration Test Suite
Sun Mar 11 14:49:02 2018
===============================================================================
server-client: protocol: transport: result:
cpp-cpp compact framed-ip success
cpp-cpp multi-binary framed-ip success
cpp-cpp compact framed-ip-ssl success
cpp-cpp multih framed-ip-ssl success
cpp-cpp multih framed-ip success
cpp-cpp multi-binary framed-ip-ssl success
cpp-cpp multih-header framed-ip success
cpp-cpp multih-header framed-ip-ssl success
cpp-cpp multij-json framed-ip-ssl success
cpp-cpp multij-json framed-ip success
cpp-cpp json framed-ip success
cpp-cpp json framed-ip-ssl success
cpp-cpp multic framed-ip success
cpp-cpp multic framed-ip-ssl success
cpp-cpp multic-compact framed-ip success
cpp-cpp multic-compact framed-ip-ssl success
cpp-cpp multij-json framed-ip-ssl success
cpp-cpp multij-json framed-ip success
cpp-cpp header framed-ip failure(-6)
cpp-cpp multij framed-ip success
cpp-cpp multij framed-ip-ssl success
cpp-cpp header framed-ip-ssl failure(-6)
cpp-cpp multic-compact framed-ip success
cpp-cpp multic-compact framed-ip-ssl success
cpp-cpp multi-binary framed-ip-ssl success
cpp-cpp multi-binary framed-ip success
cpp-cpp multi framed-ip-ssl success
cpp-cpp multi framed-ip success
cpp-cpp multih-header framed-ip-ssl success
cpp-cpp multih-header framed-ip success
cpp-cpp binary framed-ip-ssl success
cpp-cpp binary framed-ip success
===============================================================================
```
(I think you can ignore the header failures)
---
[GitHub] thrift pull request #1497: Do not call workSocket() in TNonblockigServer wit...
Posted by bgedik <gi...@git.apache.org>.
Github user bgedik commented on a diff in the pull request:
https://github.com/apache/thrift/pull/1497#discussion_r172754060
--- Diff: lib/cpp/src/thrift/server/TNonblockingServer.cpp ---
@@ -472,6 +472,18 @@ void TNonblockingServer::TConnection::workSocket() {
}
// size known; now get the rest of the frame
transition();
+
+ // If the socket has more data than the frame header, continue to work on it. This is not strictly necessary for
+ // regular sockets, because if there is more data, libevent will fire the event handler registered for read
+ // readiness, which will in turn call workSocket(). However, some socket types (such as TSSLSocket) may have the
+ // data sitting in their internal buffers and from libevent's perspective, there is no further data available. In
+ // that case, not having this workSocket() call here would result in a hang as we will never get to work the socket,
+ // despite having more data.
+ if (tSocket_->hasPendingDataToRead())
--- End diff --
It can't. Note that the line above is ``transition()``. So the next time we call ``workSocket()`` it is a different state.
Btw, I assume you notice that the original ``workSocket`` call I removed, which was causing the superfluous exception, was inside ``workSocket`` as well.
---
[GitHub] thrift pull request #1497: Do not call workSocket() in TNonblockigServer wit...
Posted by jeking3 <gi...@git.apache.org>.
Github user jeking3 commented on a diff in the pull request:
https://github.com/apache/thrift/pull/1497#discussion_r172694884
--- Diff: lib/cpp/src/thrift/server/TNonblockingServer.cpp ---
@@ -472,6 +472,18 @@ void TNonblockingServer::TConnection::workSocket() {
}
// size known; now get the rest of the frame
transition();
+
+ // If the socket has more data than the frame header, continue to work on it. This is not strictly necessary for
+ // regular sockets, because if there is more data, libevent will fire the event handler registered for read
+ // readiness, which will in turn call workSocket(). However, some socket types (such as TSSLSocket) may have the
+ // data sitting in their internal buffers and from libevent's perspective, there is no further data available. In
+ // that case, not having this workSocket() call here would result in a hang as we will never get to work the socket,
+ // despite having more data.
+ if (tSocket_->hasPendingDataToRead())
--- End diff --
Should this be an "if" or a "while"?
---
[GitHub] thrift issue #1497: Do not call workSocket() in TNonblockigServer without en...
Posted by jeking3 <gi...@git.apache.org>.
Github user jeking3 commented on the issue:
https://github.com/apache/thrift/pull/1497
Rebase on master and let's see what happens, thanks.
---
[GitHub] thrift pull request #1497: Do not call workSocket() in TNonblockigServer wit...
Posted by bgedik <gi...@git.apache.org>.
Github user bgedik commented on a diff in the pull request:
https://github.com/apache/thrift/pull/1497#discussion_r172752961
--- Diff: lib/cpp/src/thrift/server/TNonblockingServer.cpp ---
@@ -472,6 +472,18 @@ void TNonblockingServer::TConnection::workSocket() {
}
// size known; now get the rest of the frame
transition();
+
+ // If the socket has more data than the frame header, continue to work on it. This is not strictly necessary for
+ // regular sockets, because if there is more data, libevent will fire the event handler registered for read
+ // readiness, which will in turn call workSocket(). However, some socket types (such as TSSLSocket) may have the
+ // data sitting in their internal buffers and from libevent's perspective, there is no further data available. In
+ // that case, not having this workSocket() call here would result in a hang as we will never get to work the socket,
+ // despite having more data.
+ if (tSocket_->hasPendingDataToRead())
--- End diff --
It should be an ``if``. At this point, we know the frame size. The next step is to read the frame data itself. And for that, there is already logic to work the socket until all the data is read. For SSL specifically, there is no concern that the SSL buffers can keep pending data forever, because when the entire data frame arrives, there won't be any incomplete SSL bytes. In any case, the code should work generically for all socket types.
---