You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by "Pavlin Radoslavov (JIRA)" <ji...@apache.org> on 2012/07/18 07:46:34 UTC

[jira] [Reopened] (THRIFT-1442) TNonblockingServer: Refactor to allow multiple IO Threads

     [ https://issues.apache.org/jira/browse/THRIFT-1442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pavlin Radoslavov reopened THRIFT-1442:
---------------------------------------


I am reopening the issue, because it breaks the case of using
TNonblockingServer in a single-thread setup (i.e., no threads), and
event_base provided by the user. In such setup, the main eventloop can
be run/loop outside Thrift, while with the original fix for THRIFT-1442
it was not possible: loop over the libevent handler must happen inside
TNonblockingServer::serve().

The provided patch does the following:

(a) Preserves the existing multiple I/O Threads semantics as implemented
   by the original fix for THRIFT-1442:
     _server = new TNonblockingServer(_processor, _service_port);
     _server->setNumIOThreads(5);
     _server->serve();    // Main workhorse function

(b) Add-back support for using TNonblockingServer as a single thread
   and eventloop outside the server:
     _server = new TNonblockingServer(_processor, _service_port);
     _server->setServerEventHandler(_my_server_event_handler);
     _server->registerEvents(_my_event_base);
    // Continue execution and loop over my_event_base elsewhere

 * Add-back support for (b), and allowing TNonblockingServer to use a socket
   that was created/set in advance:
     _server = new TNonblockingServer(_processor, _service_port);
     _server->listenSocket(_socket);
     _server->registerEvents(_my_event_base);
    // Continue execution and loop over my_event_base elsewhere

                
> TNonblockingServer: Refactor to allow multiple IO Threads
> ---------------------------------------------------------
>
>                 Key: THRIFT-1442
>                 URL: https://issues.apache.org/jira/browse/THRIFT-1442
>             Project: Thrift
>          Issue Type: Improvement
>          Components: C++ - Library
>            Reporter: Dave Watson
>            Assignee: Dave Watson
>            Priority: Minor
>             Fix For: 0.9
>
>         Attachments: 0001-TNonblockingServer-Refactor-to-allow-multiple-IO-thr.patch, nonblocking_unsigned.patch
>
>
> From 04fc8cbc24c64e1b68a23a1df2c46056785c269d Mon Sep 17 00:00:00 2001
> From: Mark Rabkin <mr...@fb.com>
> Date: Tue, 18 May 2010 22:16:09 +0000
> Subject: [PATCH 01/56] TNonblockingServer: Refactor to allow multiple IO
>  threads, not just one
> davejwatson: This diff ads multiple IO threads to TNonblocking Server. 
> We use it extensively, it's pretty well tested other than merge errors.  
> This diff reverts THRIFT-1184, allowing re-use of an existing
> event base.  With multiple IO threads, re-using a single event base doesn't
> make sense, so this seemed ok
> Summary:
> The diff creates multiple IO threads at startup -- the number of threads in
> this diff is fixed at server start and cannot change for simplicity.  The
> first thread (id = 0) also doubles as the listen/accept thread, so there is
> still only a single thread doing accepts.  The other IO threads listen only
> on their notification pipe and the actual connection sockets.
> Also, for simplicity, each accepted connection is simply assigned in a
> round-robin fashion to the next IO thread and lives on that IO thread
> permanently.
> Note that there are only trivial changes to TConnection to get it to work, so
> most of the tricky state transitions and buffer management are unchanged.
> There is still a single worker pool for processing tasks, so that code is
> unchanged as well.
> The trickiest part of the diff requiring the most careful review is the new
> synchronization code in the TNonblockingServer to manage the connection stack
> and counters of active/inactive connections.  We now lock a mutex when
> incrementing/decrementing server counters, which is less than ideal for
> extremely high-QPS servers -- should I switch to atomic ops?
> One important change here is that while connections are created and initialized
> by the listen thread (IO thread #0), they may now be assigned to an event_base
> owned by a different IO thread.  To work safely, TConnection::init() no longer
> calls setFlags() - instead, it immediately calls TConnection::notifyIOThread().
> This results in a notification-fd event in the correct event base, which then
> calls TConnection::transition() which sets up correct read flags.  This means
> that a TConnection now calls notifyIOThread() once upon creation and assignment
> to its event_base, and thereafter after each processing call completes.
> TNonblocking server: Allow high-priority scheduler for IO threads
> Summary:
> Adds a boolean option to TNonblockingServer to use sched_setscheduler() to set
> a high scheduling priority for the IO thread(s) -- this is a POSIX api and
> should be safe on most operating systems.
> Please let me know if this is a known terrible idea, but we're experimenting
> to see if this helps the situation where we have 40 worker threads and 1 IO
> thread and the IO thread doesn't get scheduled nearly often enough.
> Reviewers: dreiss,edhall,putivsky
> Test Plan:  Need to work with Ed to run his capacity-loadtesting scripts to verify performance.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira