You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by bgedik <gi...@git.apache.org> on 2018/02/19 20:07:21 UTC

[GitHub] thrift pull request #1497: Remove workSocket call that is too early

GitHub user bgedik opened a pull request:

    https://github.com/apache/thrift/pull/1497

    Remove workSocket call that is too early

    Make the work socket conditional
    Revert back the changes
    Only call workSocket() when there is pending data to read
    Minor update to a comment
    Only call workSocket() when there is pending data to read
    Only call workSocket() when there is pending data to read
    Fix the CMake build
    Fix the CMake build
    Fix the CMake build
    Use the correct type for ioctlsocket call
    Make non-blocking peek optional
    Revert "Make non-blocking peek optional"
    This reverts commit 501c4401ddb93cf9bf451dd289d4c3768c0a7b15.
    Review fixes
    Review fixes
    Review fixes

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/bgedik/thrift bugfix/non-blocking-server-calls-workSocket-too-early

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/thrift/pull/1497.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1497
    
----
commit 6665029561d40d5b89b7ee9cb2487f6c905b904a
Author: Bugra Gedik <bg...@...>
Date:   2018-01-21T17:43:49Z

    remove workSocket call that is too early
    
    Make the work socket conditional
    
    Revert back the changes
    
    Only call workSocket() when there is pending data to read
    
    Minor update to a comment
    
    Only call workSocket() when there is pending data to read
    
    Only call workSocket() when there is pending data to read
    
    Fix the CMake build
    
    Fix the CMake build
    
    Fix the CMake build
    
    Use the correct type for ioctlsocket call
    
    Make non-blocking peek optional
    
    Revert "Make non-blocking peek optional"
    
    This reverts commit 501c4401ddb93cf9bf451dd289d4c3768c0a7b15.
    
    Review fixes
    
    Review fixes
    
    Review fixes

----


---

[GitHub] thrift issue #1497: Do not call workSocket() in TNonblockigServer without en...

Posted by bgedik <gi...@git.apache.org>.
Github user bgedik commented on the issue:

    https://github.com/apache/thrift/pull/1497
  
    @jeking3 The ``client_pool_tests`` failed in 2 of the bots within Travis-CI. I don't think it is related to my change. Are these known to fail every now and then?
    
    ```
    thrift.transport.base.TTransportException@src/thrift/transport/socket.d(255): Failed to connect to 127.0.0.1:9090.
    ----------------
    ??:? void thrift.transport.socket.TSocket.open() [0x29e331ca]
    ??:? int thrift.codegen.client_pool.TClientPool!(client_pool_test.ExTestService).TClientPool.executeOnPool!(int).executeOnPool(scope int delegate(thrift.codegen.client.TClientBase!(client_pool_test.ExTestService).TClientBase)) [0x29e204ca]
    ??:? int thrift.codegen.client_pool.TClientPool!(client_pool_test.ExTestService).TClientPool.getPort() [0x29e08916]
    ??:? void client_pool_test.syncClientPoolTest(const(ushort)[], client_pool_test.ExTestHandler[]) [0x29df090a]
    ??:? _Dmain [0x29df077b]
    thrift.base.TCompoundOperationException@src/thrift/transport/socket.d(256): All addresses tried failed (127.0.0.1:9090: "Unable to connect socket: Connection refused", 127.0.0.1:9090: "Unable to connect socket: Connection refused", 127.0.0.1:9090: "Unable to connect socket: Connection refused").
    FAIL: client_pool_test
    ```


---

[GitHub] thrift issue #1497: Do not call workSocket() in TNonblockigServer without en...

Posted by bgedik <gi...@git.apache.org>.
Github user bgedik commented on the issue:

    https://github.com/apache/thrift/pull/1497
  
    @jeking3 There is already a Jira item: https://issues.apache.org/jira/browse/THRIFT-4465
    
    Btw, it makes sense that the fix resolves the framed transport failures. Because the bug was about the transition from the frame size reading to the frame content reading.


---

[GitHub] thrift issue #1497: Do not call workSocket() in TNonblockigServer without en...

Posted by jeking3 <gi...@git.apache.org>.
Github user jeking3 commented on the issue:

    https://github.com/apache/thrift/pull/1497
  
    Wow, haven't seen a green build in a while.  Of course, none of the cross tests exercise nonblocking code. :(  Would be nice to add this as another matrix option.


---

[GitHub] thrift issue #1497: Do not call workSocket() in TNonblockigServer without en...

Posted by jeking3 <gi...@git.apache.org>.
Github user jeking3 commented on the issue:

    https://github.com/apache/thrift/pull/1497
  
    Cool so that last test run was without the fix (I thought it had it, but it only had my fixes to TestServer.cpp), and adding your fix seems to have taken care of the header failures somehow:
    ```
    root@1dcb0ce83732:/thrift/src# test/test.py --server cpp --client cpp --regex='.*framed.*-ip.*'
    Apache Thrift - Integration Test Suite
    Sun Mar 11 19:39:18 2018
    ===============================================================================
    server-client:          protocol:         transport:               result:
    cpp-cpp                 compact           framed-ip                success
    cpp-cpp                 multi-binary      framed-ip-ssl            success
    cpp-cpp                 compact           framed-ip-ssl            success
    cpp-cpp                 multi-binary      framed-ip                success
    cpp-cpp                 multih            framed-ip                success
    cpp-cpp                 multih            framed-ip-ssl            success
    cpp-cpp                 multih-header     framed-ip-ssl            success
    cpp-cpp                 multih-header     framed-ip                success
    cpp-cpp                 multij-json       framed-ip-ssl            success
    cpp-cpp                 multij-json       framed-ip                success
    cpp-cpp                 json              framed-ip-ssl            success
    cpp-cpp                 json              framed-ip                success
    cpp-cpp                 multic            framed-ip                success
    cpp-cpp                 multic            framed-ip-ssl            success
    cpp-cpp                 multic-compact    framed-ip-ssl            success
    cpp-cpp                 multic-compact    framed-ip                success
    cpp-cpp                 multij-json       framed-ip-ssl            success
    cpp-cpp                 multij-json       framed-ip                success
    cpp-cpp                 multij            framed-ip-ssl            success
    cpp-cpp                 multij            framed-ip                success
    cpp-cpp                 header            framed-ip-ssl            success
    cpp-cpp                 header            framed-ip                success
    cpp-cpp                 multic-compact    framed-ip-ssl            success
    cpp-cpp                 multic-compact    framed-ip                success
    cpp-cpp                 multi-binary      framed-ip-ssl            success
    cpp-cpp                 multi-binary      framed-ip                success
    cpp-cpp                 multi             framed-ip                success
    cpp-cpp                 multi             framed-ip-ssl            success
    cpp-cpp                 multih-header     framed-ip-ssl            success
    cpp-cpp                 multih-header     framed-ip                success
    cpp-cpp                 binary            framed-ip                success
    cpp-cpp                 binary            framed-ip-ssl            success
    ===============================================================================
    No unexpected failures.
    ```


---

[GitHub] thrift issue #1497: Do not call workSocket() in TNonblockigServer without en...

Posted by ozymaxx <gi...@git.apache.org>.
Github user ozymaxx commented on the issue:

    https://github.com/apache/thrift/pull/1497
  
    I also encounter the same error, but it does not show up every time I run the tests on Travis CI.


---

[GitHub] thrift issue #1497: Do not call workSocket() in TNonblockigServer without en...

Posted by jeking3 <gi...@git.apache.org>.
Github user jeking3 commented on the issue:

    https://github.com/apache/thrift/pull/1497
  
    I'm not aware of those being common or known.  I have a bit of work to do in order to stabilize the CI builds and get them back to green...


---

[GitHub] thrift issue #1497: Do not call workSocket() in TNonblockigServer without en...

Posted by bgedik <gi...@git.apache.org>.
Github user bgedik commented on the issue:

    https://github.com/apache/thrift/pull/1497
  
    @jeking3 I think all tests have passed.


---

[GitHub] thrift pull request #1497: Do not call workSocket() in TNonblockigServer wit...

Posted by jeking3 <gi...@git.apache.org>.
Github user jeking3 commented on a diff in the pull request:

    https://github.com/apache/thrift/pull/1497#discussion_r172695014
  
    --- Diff: lib/cpp/src/thrift/server/TNonblockingServer.cpp ---
    @@ -472,6 +472,18 @@ void TNonblockingServer::TConnection::workSocket() {
         }
         // size known; now get the rest of the frame
         transition();
    +
    +    // If the socket has more data than the frame header, continue to work on it. This is not strictly necessary for
    +    // regular sockets, because if there is more data, libevent will fire the event handler registered for read
    +    // readiness, which will in turn call workSocket(). However, some socket types (such as TSSLSocket) may have the
    +    // data sitting in their internal buffers and from libevent's perspective, there is no further data available. In
    +    // that case, not having this workSocket() call here would result in a hang as we will never get to work the socket,
    +    // despite having more data.
    +    if (tSocket_->hasPendingDataToRead())
    --- End diff --
    
    I am concerned this can cause an infinite loop if data is always available, since it calls itself.


---

[GitHub] thrift issue #1497: Do not call workSocket() in TNonblockigServer without en...

Posted by jeking3 <gi...@git.apache.org>.
Github user jeking3 commented on the issue:

    https://github.com/apache/thrift/pull/1497
  
    What's the Jira ticket ID for this work?


---

[GitHub] thrift pull request #1497: Do not call workSocket() in TNonblockigServer wit...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/thrift/pull/1497


---

[GitHub] thrift issue #1497: Do not call workSocket() in TNonblockigServer without en...

Posted by jeking3 <gi...@git.apache.org>.
Github user jeking3 commented on the issue:

    https://github.com/apache/thrift/pull/1497
  
    You need to create a Jira ticket in the THRIFT project (https://issues.apache.org/jira/projects/THRIFT/issues) that describes the bug and resolution so that I can merge it.
    
    ---
    
    I was able to test these changes out by making some changes to the cpp TestServer for crosstest which I will check in today.  I changed my local tests.json file to add "--server-type=nonblocking" to each cpp server and changed some code to make additional cases work:
    
    ```
    root@8d560c4cc0fa:/thrift/src# test/test.py --server cpp --client cpp --regex='.*framed.*-ip.*'
    Apache Thrift - Integration Test Suite
    Sun Mar 11 14:49:02 2018
    ===============================================================================
    server-client:          protocol:         transport:               result:
    cpp-cpp                 compact           framed-ip                success
    cpp-cpp                 multi-binary      framed-ip                success
    cpp-cpp                 compact           framed-ip-ssl            success
    cpp-cpp                 multih            framed-ip-ssl            success
    cpp-cpp                 multih            framed-ip                success
    cpp-cpp                 multi-binary      framed-ip-ssl            success
    cpp-cpp                 multih-header     framed-ip                success
    cpp-cpp                 multih-header     framed-ip-ssl            success
    cpp-cpp                 multij-json       framed-ip-ssl            success
    cpp-cpp                 multij-json       framed-ip                success
    cpp-cpp                 json              framed-ip                success
    cpp-cpp                 json              framed-ip-ssl            success
    cpp-cpp                 multic            framed-ip                success
    cpp-cpp                 multic            framed-ip-ssl            success
    cpp-cpp                 multic-compact    framed-ip                success
    cpp-cpp                 multic-compact    framed-ip-ssl            success
    cpp-cpp                 multij-json       framed-ip-ssl            success
    cpp-cpp                 multij-json       framed-ip                success
    cpp-cpp                 header            framed-ip                failure(-6)
    cpp-cpp                 multij            framed-ip                success
    cpp-cpp                 multij            framed-ip-ssl            success
    cpp-cpp                 header            framed-ip-ssl            failure(-6)
    cpp-cpp                 multic-compact    framed-ip                success
    cpp-cpp                 multic-compact    framed-ip-ssl            success
    cpp-cpp                 multi-binary      framed-ip-ssl            success
    cpp-cpp                 multi-binary      framed-ip                success
    cpp-cpp                 multi             framed-ip-ssl            success
    cpp-cpp                 multi             framed-ip                success
    cpp-cpp                 multih-header     framed-ip-ssl            success
    cpp-cpp                 multih-header     framed-ip                success
    cpp-cpp                 binary            framed-ip-ssl            success
    cpp-cpp                 binary            framed-ip                success
    ===============================================================================
    ```
    (I think you can ignore the header failures)


---

[GitHub] thrift pull request #1497: Do not call workSocket() in TNonblockigServer wit...

Posted by bgedik <gi...@git.apache.org>.
Github user bgedik commented on a diff in the pull request:

    https://github.com/apache/thrift/pull/1497#discussion_r172754060
  
    --- Diff: lib/cpp/src/thrift/server/TNonblockingServer.cpp ---
    @@ -472,6 +472,18 @@ void TNonblockingServer::TConnection::workSocket() {
         }
         // size known; now get the rest of the frame
         transition();
    +
    +    // If the socket has more data than the frame header, continue to work on it. This is not strictly necessary for
    +    // regular sockets, because if there is more data, libevent will fire the event handler registered for read
    +    // readiness, which will in turn call workSocket(). However, some socket types (such as TSSLSocket) may have the
    +    // data sitting in their internal buffers and from libevent's perspective, there is no further data available. In
    +    // that case, not having this workSocket() call here would result in a hang as we will never get to work the socket,
    +    // despite having more data.
    +    if (tSocket_->hasPendingDataToRead())
    --- End diff --
    
    It can't. Note that the line above is ``transition()``. So the next time we call ``workSocket()`` it is a different state. 
    
    Btw, I assume you notice that the original ``workSocket`` call I removed, which was causing the superfluous exception, was inside ``workSocket`` as well.


---

[GitHub] thrift pull request #1497: Do not call workSocket() in TNonblockigServer wit...

Posted by jeking3 <gi...@git.apache.org>.
Github user jeking3 commented on a diff in the pull request:

    https://github.com/apache/thrift/pull/1497#discussion_r172694884
  
    --- Diff: lib/cpp/src/thrift/server/TNonblockingServer.cpp ---
    @@ -472,6 +472,18 @@ void TNonblockingServer::TConnection::workSocket() {
         }
         // size known; now get the rest of the frame
         transition();
    +
    +    // If the socket has more data than the frame header, continue to work on it. This is not strictly necessary for
    +    // regular sockets, because if there is more data, libevent will fire the event handler registered for read
    +    // readiness, which will in turn call workSocket(). However, some socket types (such as TSSLSocket) may have the
    +    // data sitting in their internal buffers and from libevent's perspective, there is no further data available. In
    +    // that case, not having this workSocket() call here would result in a hang as we will never get to work the socket,
    +    // despite having more data.
    +    if (tSocket_->hasPendingDataToRead())
    --- End diff --
    
    Should this be an "if" or a "while"?


---

[GitHub] thrift issue #1497: Do not call workSocket() in TNonblockigServer without en...

Posted by jeking3 <gi...@git.apache.org>.
Github user jeking3 commented on the issue:

    https://github.com/apache/thrift/pull/1497
  
    Rebase on master and let's see what happens, thanks.


---

[GitHub] thrift pull request #1497: Do not call workSocket() in TNonblockigServer wit...

Posted by bgedik <gi...@git.apache.org>.
Github user bgedik commented on a diff in the pull request:

    https://github.com/apache/thrift/pull/1497#discussion_r172752961
  
    --- Diff: lib/cpp/src/thrift/server/TNonblockingServer.cpp ---
    @@ -472,6 +472,18 @@ void TNonblockingServer::TConnection::workSocket() {
         }
         // size known; now get the rest of the frame
         transition();
    +
    +    // If the socket has more data than the frame header, continue to work on it. This is not strictly necessary for
    +    // regular sockets, because if there is more data, libevent will fire the event handler registered for read
    +    // readiness, which will in turn call workSocket(). However, some socket types (such as TSSLSocket) may have the
    +    // data sitting in their internal buffers and from libevent's perspective, there is no further data available. In
    +    // that case, not having this workSocket() call here would result in a hang as we will never get to work the socket,
    +    // despite having more data.
    +    if (tSocket_->hasPendingDataToRead())
    --- End diff --
    
    It should be an ``if``. At this point, we know the frame size. The next step is to read the frame data itself. And for that, there is already logic to work the socket until all the data is read. For SSL specifically, there is no concern that the SSL buffers can keep pending data forever, because when the entire data frame arrives, there won't be any incomplete SSL bytes. In any case, the code should work generically for all socket types.
    



---