You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by "Dvir Volk (JIRA)" <ji...@apache.org> on 2011/03/03 23:51:43 UTC
[jira] Created: (THRIFT-1083) Preforking python process pool server
Preforking python process pool server
-------------------------------------
Key: THRIFT-1083
URL: https://issues.apache.org/jira/browse/THRIFT-1083
Project: Thrift
Issue Type: New Feature
Components: Python - Library
Affects Versions: 0.6
Environment: linux. haven't tested on windows
Reporter: Dvir Volk
Priority: Minor
This patch adds a new server to the python library: TProcessPoolServer, which is a preforking python server.
this server is sort of a mix between TForkingServer and TThreadPoolServer: It has a pool of workers that are preforked subprocesses.
This approach allows the user to avoid the python GIL single processor limit in threading applications, and use a high performance server, and at the same time it avoids the need to fork a child process for each connection, as is the case with the forking server.
I've benchmarked it to be about 5-6 times faster than TThreadPoolServer on a quad Corei7 CPU, and about the same amount faster than TForkingServer if you are not using persistent connections and forking a child for each request.
The patch also updates the python unit tests to include tests for this server.
Notes:
1. Of course this server has the limitations of forking regarding shared state and memory copying.
2. You should NOT kill the server with kill -9 - as this will not allow the parent process to terminate its children, resulting in orphaned processes that keep your socket open. either run stop() in your app or kill the process with SIGINT. It will respond to ctrl+C however
3. use setNumWorkers(n) before starting the server to determine how many processes you want to spawn. the default is 10.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (THRIFT-1083) Preforking python process pool
server
Posted by "David Reiss (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004370#comment-13004370 ]
David Reiss commented on THRIFT-1083:
-------------------------------------
Looks fine to me.
> Preforking python process pool server
> -------------------------------------
>
> Key: THRIFT-1083
> URL: https://issues.apache.org/jira/browse/THRIFT-1083
> Project: Thrift
> Issue Type: New Feature
> Components: Python - Library
> Affects Versions: 0.6
> Environment: linux. haven't tested on windows
> Reporter: Dvir Volk
> Priority: Minor
> Attachments: TProcessPoolServer.patch, TProcessPoolServer.py, TProcessPoolServer.py
>
>
> This patch adds a new server to the python library: TProcessPoolServer, which is a preforking python server.
> this server is sort of a mix between TForkingServer and TThreadPoolServer: It has a pool of workers that are preforked subprocesses.
> This approach allows the user to avoid the python GIL single processor limit in threading applications, and use a high performance server, and at the same time it avoids the need to fork a child process for each connection, as is the case with the forking server.
> I've benchmarked it to be about 5-6 times faster than TThreadPoolServer on a quad Corei7 CPU, and about the same amount faster than TForkingServer if you are not using persistent connections and forking a child for each request.
> The patch also updates the python unit tests to include tests for this server.
> Notes:
> 1. Of course this server has the limitations of forking regarding shared state and memory copying.
> 2. You should NOT kill the server with kill -9 - as this will not allow the parent process to terminate its children, resulting in orphaned processes that keep your socket open. either run stop() in your app or kill the process with SIGINT. It will respond to ctrl+C however
> 3. use setNumWorkers(n) before starting the server to determine how many processes you want to spawn. the default is 10.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (THRIFT-1083) Preforking python process pool server
Posted by "Dvir Volk (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dvir Volk updated THRIFT-1083:
------------------------------
Attachment: TProcessPoolServer.py
This is a newer version of the server, that adds an optional "post fork callback" that allows you to initialize state in individual handler processes, after the handlers have been forked. I used it for example to init redis connections.
> Preforking python process pool server
> -------------------------------------
>
> Key: THRIFT-1083
> URL: https://issues.apache.org/jira/browse/THRIFT-1083
> Project: Thrift
> Issue Type: New Feature
> Components: Python - Library
> Affects Versions: 0.6
> Environment: linux. haven't tested on windows
> Reporter: Dvir Volk
> Priority: Minor
> Attachments: TProcessPoolServer.patch, TProcessPoolServer.py, TProcessPoolServer.py
>
>
> This patch adds a new server to the python library: TProcessPoolServer, which is a preforking python server.
> this server is sort of a mix between TForkingServer and TThreadPoolServer: It has a pool of workers that are preforked subprocesses.
> This approach allows the user to avoid the python GIL single processor limit in threading applications, and use a high performance server, and at the same time it avoids the need to fork a child process for each connection, as is the case with the forking server.
> I've benchmarked it to be about 5-6 times faster than TThreadPoolServer on a quad Corei7 CPU, and about the same amount faster than TForkingServer if you are not using persistent connections and forking a child for each request.
> The patch also updates the python unit tests to include tests for this server.
> Notes:
> 1. Of course this server has the limitations of forking regarding shared state and memory copying.
> 2. You should NOT kill the server with kill -9 - as this will not allow the parent process to terminate its children, resulting in orphaned processes that keep your socket open. either run stop() in your app or kill the process with SIGINT. It will respond to ctrl+C however
> 3. use setNumWorkers(n) before starting the server to determine how many processes you want to spawn. the default is 10.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Closed: (THRIFT-1083) Preforking python process pool server
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury closed THRIFT-1083.
---------------------------------
Resolution: Fixed
Fix Version/s: 0.7
Assignee: Dvir Volk
I just committed the latest version of the server. Thanks for the contribution, Dvir!
> Preforking python process pool server
> -------------------------------------
>
> Key: THRIFT-1083
> URL: https://issues.apache.org/jira/browse/THRIFT-1083
> Project: Thrift
> Issue Type: New Feature
> Components: Python - Library
> Affects Versions: 0.6
> Environment: linux. haven't tested on windows
> Reporter: Dvir Volk
> Assignee: Dvir Volk
> Priority: Minor
> Fix For: 0.7
>
> Attachments: TProcessPoolServer.patch, TProcessPoolServer.py, TProcessPoolServer.py
>
>
> This patch adds a new server to the python library: TProcessPoolServer, which is a preforking python server.
> this server is sort of a mix between TForkingServer and TThreadPoolServer: It has a pool of workers that are preforked subprocesses.
> This approach allows the user to avoid the python GIL single processor limit in threading applications, and use a high performance server, and at the same time it avoids the need to fork a child process for each connection, as is the case with the forking server.
> I've benchmarked it to be about 5-6 times faster than TThreadPoolServer on a quad Corei7 CPU, and about the same amount faster than TForkingServer if you are not using persistent connections and forking a child for each request.
> The patch also updates the python unit tests to include tests for this server.
> Notes:
> 1. Of course this server has the limitations of forking regarding shared state and memory copying.
> 2. You should NOT kill the server with kill -9 - as this will not allow the parent process to terminate its children, resulting in orphaned processes that keep your socket open. either run stop() in your app or kill the process with SIGINT. It will respond to ctrl+C however
> 3. use setNumWorkers(n) before starting the server to determine how many processes you want to spawn. the default is 10.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (THRIFT-1083) Preforking python process pool
server
Posted by "Jake Farrell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003545#comment-13003545 ]
Jake Farrell commented on THRIFT-1083:
--------------------------------------
TProcessPoolServer looks good, runs as expected. Some unnecessary spacing in patch (lines 86, 100, 101, 150), but good otherwise
> Preforking python process pool server
> -------------------------------------
>
> Key: THRIFT-1083
> URL: https://issues.apache.org/jira/browse/THRIFT-1083
> Project: Thrift
> Issue Type: New Feature
> Components: Python - Library
> Affects Versions: 0.6
> Environment: linux. haven't tested on windows
> Reporter: Dvir Volk
> Priority: Minor
> Attachments: TProcessPoolServer.patch, TProcessPoolServer.py
>
>
> This patch adds a new server to the python library: TProcessPoolServer, which is a preforking python server.
> this server is sort of a mix between TForkingServer and TThreadPoolServer: It has a pool of workers that are preforked subprocesses.
> This approach allows the user to avoid the python GIL single processor limit in threading applications, and use a high performance server, and at the same time it avoids the need to fork a child process for each connection, as is the case with the forking server.
> I've benchmarked it to be about 5-6 times faster than TThreadPoolServer on a quad Corei7 CPU, and about the same amount faster than TForkingServer if you are not using persistent connections and forking a child for each request.
> The patch also updates the python unit tests to include tests for this server.
> Notes:
> 1. Of course this server has the limitations of forking regarding shared state and memory copying.
> 2. You should NOT kill the server with kill -9 - as this will not allow the parent process to terminate its children, resulting in orphaned processes that keep your socket open. either run stop() in your app or kill the process with SIGINT. It will respond to ctrl+C however
> 3. use setNumWorkers(n) before starting the server to determine how many processes you want to spawn. the default is 10.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (THRIFT-1083) Preforking python process pool
server
Posted by "Dvir Volk (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281615#comment-13281615 ]
Dvir Volk commented on THRIFT-1083:
-----------------------------------
@Jonathan, I'll work on it tonight. mind opening another issue for it?
> Preforking python process pool server
> -------------------------------------
>
> Key: THRIFT-1083
> URL: https://issues.apache.org/jira/browse/THRIFT-1083
> Project: Thrift
> Issue Type: New Feature
> Components: Python - Library
> Affects Versions: 0.6
> Environment: linux. haven't tested on windows
> Reporter: Dvir Volk
> Assignee: Dvir Volk
> Priority: Minor
> Fix For: 0.7
>
> Attachments: TProcessPoolServer.patch, TProcessPoolServer.py, TProcessPoolServer.py
>
>
> This patch adds a new server to the python library: TProcessPoolServer, which is a preforking python server.
> this server is sort of a mix between TForkingServer and TThreadPoolServer: It has a pool of workers that are preforked subprocesses.
> This approach allows the user to avoid the python GIL single processor limit in threading applications, and use a high performance server, and at the same time it avoids the need to fork a child process for each connection, as is the case with the forking server.
> I've benchmarked it to be about 5-6 times faster than TThreadPoolServer on a quad Corei7 CPU, and about the same amount faster than TForkingServer if you are not using persistent connections and forking a child for each request.
> The patch also updates the python unit tests to include tests for this server.
> Notes:
> 1. Of course this server has the limitations of forking regarding shared state and memory copying.
> 2. You should NOT kill the server with kill -9 - as this will not allow the parent process to terminate its children, resulting in orphaned processes that keep your socket open. either run stop() in your app or kill the process with SIGINT. It will respond to ctrl+C however
> 3. use setNumWorkers(n) before starting the server to determine how many processes you want to spawn. the default is 10.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (THRIFT-1083) Preforking python process pool
server
Posted by "Dvir Volk (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280523#comment-13280523 ]
Dvir Volk commented on THRIFT-1083:
-----------------------------------
@Jonathan Hi,
The tests currently in the trunk are not the tests I originally wrote for the server so I'll need to play with them to see what has changed and why.
I have a newer, richer version of the server I can share - it has post fork and on terminate callbacks for workers, and better termination code.
> Preforking python process pool server
> -------------------------------------
>
> Key: THRIFT-1083
> URL: https://issues.apache.org/jira/browse/THRIFT-1083
> Project: Thrift
> Issue Type: New Feature
> Components: Python - Library
> Affects Versions: 0.6
> Environment: linux. haven't tested on windows
> Reporter: Dvir Volk
> Assignee: Dvir Volk
> Priority: Minor
> Fix For: 0.7
>
> Attachments: TProcessPoolServer.patch, TProcessPoolServer.py, TProcessPoolServer.py
>
>
> This patch adds a new server to the python library: TProcessPoolServer, which is a preforking python server.
> this server is sort of a mix between TForkingServer and TThreadPoolServer: It has a pool of workers that are preforked subprocesses.
> This approach allows the user to avoid the python GIL single processor limit in threading applications, and use a high performance server, and at the same time it avoids the need to fork a child process for each connection, as is the case with the forking server.
> I've benchmarked it to be about 5-6 times faster than TThreadPoolServer on a quad Corei7 CPU, and about the same amount faster than TForkingServer if you are not using persistent connections and forking a child for each request.
> The patch also updates the python unit tests to include tests for this server.
> Notes:
> 1. Of course this server has the limitations of forking regarding shared state and memory copying.
> 2. You should NOT kill the server with kill -9 - as this will not allow the parent process to terminate its children, resulting in orphaned processes that keep your socket open. either run stop() in your app or kill the process with SIGINT. It will respond to ctrl+C however
> 3. use setNumWorkers(n) before starting the server to determine how many processes you want to spawn. the default is 10.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (THRIFT-1083) Preforking python process pool
server
Posted by "Dvir Volk (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280566#comment-13280566 ]
Dvir Volk commented on THRIFT-1083:
-----------------------------------
@jonathan, here you go. I ran some of the tests with it and it seems to work fine. I actually think it has a bit of redundant voodoo for stopping processes, but it works, so you know the rule... :)
http://dl.dropbox.com/u/12876951/TProcessPoolServer.py
> Preforking python process pool server
> -------------------------------------
>
> Key: THRIFT-1083
> URL: https://issues.apache.org/jira/browse/THRIFT-1083
> Project: Thrift
> Issue Type: New Feature
> Components: Python - Library
> Affects Versions: 0.6
> Environment: linux. haven't tested on windows
> Reporter: Dvir Volk
> Assignee: Dvir Volk
> Priority: Minor
> Fix For: 0.7
>
> Attachments: TProcessPoolServer.patch, TProcessPoolServer.py, TProcessPoolServer.py
>
>
> This patch adds a new server to the python library: TProcessPoolServer, which is a preforking python server.
> this server is sort of a mix between TForkingServer and TThreadPoolServer: It has a pool of workers that are preforked subprocesses.
> This approach allows the user to avoid the python GIL single processor limit in threading applications, and use a high performance server, and at the same time it avoids the need to fork a child process for each connection, as is the case with the forking server.
> I've benchmarked it to be about 5-6 times faster than TThreadPoolServer on a quad Corei7 CPU, and about the same amount faster than TForkingServer if you are not using persistent connections and forking a child for each request.
> The patch also updates the python unit tests to include tests for this server.
> Notes:
> 1. Of course this server has the limitations of forking regarding shared state and memory copying.
> 2. You should NOT kill the server with kill -9 - as this will not allow the parent process to terminate its children, resulting in orphaned processes that keep your socket open. either run stop() in your app or kill the process with SIGINT. It will respond to ctrl+C however
> 3. use setNumWorkers(n) before starting the server to determine how many processes you want to spawn. the default is 10.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (THRIFT-1083) Preforking python process pool
server
Posted by "Jonathan Kaczynski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13281179#comment-13281179 ]
Jonathan Kaczynski commented on THRIFT-1083:
--------------------------------------------
Thanks again Dvir. Here's how I'm using the server: https://gist.github.com/2771144
I'm sending the parent process the TERM signal, and noticed two issues:
1. The child processes won't terminate until they have attempted to field a request. If server.stop() is called before it has fielded a request, the child process will continue running until it services a request and then it will shutdown.
2. The parent process never shits down.
> Preforking python process pool server
> -------------------------------------
>
> Key: THRIFT-1083
> URL: https://issues.apache.org/jira/browse/THRIFT-1083
> Project: Thrift
> Issue Type: New Feature
> Components: Python - Library
> Affects Versions: 0.6
> Environment: linux. haven't tested on windows
> Reporter: Dvir Volk
> Assignee: Dvir Volk
> Priority: Minor
> Fix For: 0.7
>
> Attachments: TProcessPoolServer.patch, TProcessPoolServer.py, TProcessPoolServer.py
>
>
> This patch adds a new server to the python library: TProcessPoolServer, which is a preforking python server.
> this server is sort of a mix between TForkingServer and TThreadPoolServer: It has a pool of workers that are preforked subprocesses.
> This approach allows the user to avoid the python GIL single processor limit in threading applications, and use a high performance server, and at the same time it avoids the need to fork a child process for each connection, as is the case with the forking server.
> I've benchmarked it to be about 5-6 times faster than TThreadPoolServer on a quad Corei7 CPU, and about the same amount faster than TForkingServer if you are not using persistent connections and forking a child for each request.
> The patch also updates the python unit tests to include tests for this server.
> Notes:
> 1. Of course this server has the limitations of forking regarding shared state and memory copying.
> 2. You should NOT kill the server with kill -9 - as this will not allow the parent process to terminate its children, resulting in orphaned processes that keep your socket open. either run stop() in your app or kill the process with SIGINT. It will respond to ctrl+C however
> 3. use setNumWorkers(n) before starting the server to determine how many processes you want to spawn. the default is 10.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (THRIFT-1083) Preforking python process pool
server
Posted by "Dvir Volk (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13003564#comment-13003564 ]
Dvir Volk commented on THRIFT-1083:
-----------------------------------
Thanks.
The one issue I've come across is the fact that if your handler class has stuff like sockets in its state, and they get initialized on its constructor (which means before forking), some of the state will be copied and some shared, but the handler's constructor of course won't be called again for the children.
I solved this by adding a post fork callback that the server calls in the handler, where you can do all this initialization (for example database connections) post forking. This is set by calling setPostForkCallback(func) to the server before running it.
I'll attach the newer version here.
> Preforking python process pool server
> -------------------------------------
>
> Key: THRIFT-1083
> URL: https://issues.apache.org/jira/browse/THRIFT-1083
> Project: Thrift
> Issue Type: New Feature
> Components: Python - Library
> Affects Versions: 0.6
> Environment: linux. haven't tested on windows
> Reporter: Dvir Volk
> Priority: Minor
> Attachments: TProcessPoolServer.patch, TProcessPoolServer.py
>
>
> This patch adds a new server to the python library: TProcessPoolServer, which is a preforking python server.
> this server is sort of a mix between TForkingServer and TThreadPoolServer: It has a pool of workers that are preforked subprocesses.
> This approach allows the user to avoid the python GIL single processor limit in threading applications, and use a high performance server, and at the same time it avoids the need to fork a child process for each connection, as is the case with the forking server.
> I've benchmarked it to be about 5-6 times faster than TThreadPoolServer on a quad Corei7 CPU, and about the same amount faster than TForkingServer if you are not using persistent connections and forking a child for each request.
> The patch also updates the python unit tests to include tests for this server.
> Notes:
> 1. Of course this server has the limitations of forking regarding shared state and memory copying.
> 2. You should NOT kill the server with kill -9 - as this will not allow the parent process to terminate its children, resulting in orphaned processes that keep your socket open. either run stop() in your app or kill the process with SIGINT. It will respond to ctrl+C however
> 3. use setNumWorkers(n) before starting the server to determine how many processes you want to spawn. the default is 10.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (THRIFT-1083) Preforking python process pool
server
Posted by "Dvir Volk (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250216#comment-13250216 ]
Dvir Volk commented on THRIFT-1083:
-----------------------------------
@goir you can add shared state when creating your handler class. That's trivial. But you'll have to use python's dedicated multiprocessing shared ctypes variables to make them accessible via all the processes. See this for an example: http://docs.python.org/library/multiprocessing.html#sharing-state-between-processes
Another option is to use the setPostForkCallback(func) call, and have the server call a function after the sub processes have spawned. This is good if you want NOT shared stuff like sockets etc.
I'm doing both strategies in my server and sharing state without a problem. I hope this answers your question.
> Preforking python process pool server
> -------------------------------------
>
> Key: THRIFT-1083
> URL: https://issues.apache.org/jira/browse/THRIFT-1083
> Project: Thrift
> Issue Type: New Feature
> Components: Python - Library
> Affects Versions: 0.6
> Environment: linux. haven't tested on windows
> Reporter: Dvir Volk
> Assignee: Dvir Volk
> Priority: Minor
> Fix For: 0.7
>
> Attachments: TProcessPoolServer.patch, TProcessPoolServer.py, TProcessPoolServer.py
>
>
> This patch adds a new server to the python library: TProcessPoolServer, which is a preforking python server.
> this server is sort of a mix between TForkingServer and TThreadPoolServer: It has a pool of workers that are preforked subprocesses.
> This approach allows the user to avoid the python GIL single processor limit in threading applications, and use a high performance server, and at the same time it avoids the need to fork a child process for each connection, as is the case with the forking server.
> I've benchmarked it to be about 5-6 times faster than TThreadPoolServer on a quad Corei7 CPU, and about the same amount faster than TForkingServer if you are not using persistent connections and forking a child for each request.
> The patch also updates the python unit tests to include tests for this server.
> Notes:
> 1. Of course this server has the limitations of forking regarding shared state and memory copying.
> 2. You should NOT kill the server with kill -9 - as this will not allow the parent process to terminate its children, resulting in orphaned processes that keep your socket open. either run stop() in your app or kill the process with SIGINT. It will respond to ctrl+C however
> 3. use setNumWorkers(n) before starting the server to determine how many processes you want to spawn. the default is 10.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (THRIFT-1083) Preforking python process pool
server
Posted by "Goir Riog (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250237#comment-13250237 ]
Goir Riog commented on THRIFT-1083:
-----------------------------------
Thanks, this works. I was thinking wayy to complicated :)
> Preforking python process pool server
> -------------------------------------
>
> Key: THRIFT-1083
> URL: https://issues.apache.org/jira/browse/THRIFT-1083
> Project: Thrift
> Issue Type: New Feature
> Components: Python - Library
> Affects Versions: 0.6
> Environment: linux. haven't tested on windows
> Reporter: Dvir Volk
> Assignee: Dvir Volk
> Priority: Minor
> Fix For: 0.7
>
> Attachments: TProcessPoolServer.patch, TProcessPoolServer.py, TProcessPoolServer.py
>
>
> This patch adds a new server to the python library: TProcessPoolServer, which is a preforking python server.
> this server is sort of a mix between TForkingServer and TThreadPoolServer: It has a pool of workers that are preforked subprocesses.
> This approach allows the user to avoid the python GIL single processor limit in threading applications, and use a high performance server, and at the same time it avoids the need to fork a child process for each connection, as is the case with the forking server.
> I've benchmarked it to be about 5-6 times faster than TThreadPoolServer on a quad Corei7 CPU, and about the same amount faster than TForkingServer if you are not using persistent connections and forking a child for each request.
> The patch also updates the python unit tests to include tests for this server.
> Notes:
> 1. Of course this server has the limitations of forking regarding shared state and memory copying.
> 2. You should NOT kill the server with kill -9 - as this will not allow the parent process to terminate its children, resulting in orphaned processes that keep your socket open. either run stop() in your app or kill the process with SIGINT. It will respond to ctrl+C however
> 3. use setNumWorkers(n) before starting the server to determine how many processes you want to spawn. the default is 10.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (THRIFT-1083) Preforking python process pool
server
Posted by "Dvir Volk (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004843#comment-13004843 ]
Dvir Volk commented on THRIFT-1083:
-----------------------------------
you're welcome, it's a pleasure to give back to such a fine project.
> Preforking python process pool server
> -------------------------------------
>
> Key: THRIFT-1083
> URL: https://issues.apache.org/jira/browse/THRIFT-1083
> Project: Thrift
> Issue Type: New Feature
> Components: Python - Library
> Affects Versions: 0.6
> Environment: linux. haven't tested on windows
> Reporter: Dvir Volk
> Assignee: Dvir Volk
> Priority: Minor
> Fix For: 0.7
>
> Attachments: TProcessPoolServer.patch, TProcessPoolServer.py, TProcessPoolServer.py
>
>
> This patch adds a new server to the python library: TProcessPoolServer, which is a preforking python server.
> this server is sort of a mix between TForkingServer and TThreadPoolServer: It has a pool of workers that are preforked subprocesses.
> This approach allows the user to avoid the python GIL single processor limit in threading applications, and use a high performance server, and at the same time it avoids the need to fork a child process for each connection, as is the case with the forking server.
> I've benchmarked it to be about 5-6 times faster than TThreadPoolServer on a quad Corei7 CPU, and about the same amount faster than TForkingServer if you are not using persistent connections and forking a child for each request.
> The patch also updates the python unit tests to include tests for this server.
> Notes:
> 1. Of course this server has the limitations of forking regarding shared state and memory copying.
> 2. You should NOT kill the server with kill -9 - as this will not allow the parent process to terminate its children, resulting in orphaned processes that keep your socket open. either run stop() in your app or kill the process with SIGINT. It will respond to ctrl+C however
> 3. use setNumWorkers(n) before starting the server to determine how many processes you want to spawn. the default is 10.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (THRIFT-1083) Preforking python process pool
server
Posted by "Jonathan Kaczynski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284890#comment-13284890 ]
Jonathan Kaczynski commented on THRIFT-1083:
--------------------------------------------
@Dvir, I've created the following tickets: THRIFT-1616, THRIFT-1617
> Preforking python process pool server
> -------------------------------------
>
> Key: THRIFT-1083
> URL: https://issues.apache.org/jira/browse/THRIFT-1083
> Project: Thrift
> Issue Type: New Feature
> Components: Python - Library
> Affects Versions: 0.6
> Environment: linux. haven't tested on windows
> Reporter: Dvir Volk
> Assignee: Dvir Volk
> Priority: Minor
> Fix For: 0.7
>
> Attachments: TProcessPoolServer.patch, TProcessPoolServer.py, TProcessPoolServer.py
>
>
> This patch adds a new server to the python library: TProcessPoolServer, which is a preforking python server.
> this server is sort of a mix between TForkingServer and TThreadPoolServer: It has a pool of workers that are preforked subprocesses.
> This approach allows the user to avoid the python GIL single processor limit in threading applications, and use a high performance server, and at the same time it avoids the need to fork a child process for each connection, as is the case with the forking server.
> I've benchmarked it to be about 5-6 times faster than TThreadPoolServer on a quad Corei7 CPU, and about the same amount faster than TForkingServer if you are not using persistent connections and forking a child for each request.
> The patch also updates the python unit tests to include tests for this server.
> Notes:
> 1. Of course this server has the limitations of forking regarding shared state and memory copying.
> 2. You should NOT kill the server with kill -9 - as this will not allow the parent process to terminate its children, resulting in orphaned processes that keep your socket open. either run stop() in your app or kill the process with SIGINT. It will respond to ctrl+C however
> 3. use setNumWorkers(n) before starting the server to determine how many processes you want to spawn. the default is 10.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (THRIFT-1083) Preforking python process pool
server
Posted by "Jonathan Kaczynski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280531#comment-13280531 ]
Jonathan Kaczynski commented on THRIFT-1083:
--------------------------------------------
@dvir, I would _really_ appreciate that. Thank you.
> Preforking python process pool server
> -------------------------------------
>
> Key: THRIFT-1083
> URL: https://issues.apache.org/jira/browse/THRIFT-1083
> Project: Thrift
> Issue Type: New Feature
> Components: Python - Library
> Affects Versions: 0.6
> Environment: linux. haven't tested on windows
> Reporter: Dvir Volk
> Assignee: Dvir Volk
> Priority: Minor
> Fix For: 0.7
>
> Attachments: TProcessPoolServer.patch, TProcessPoolServer.py, TProcessPoolServer.py
>
>
> This patch adds a new server to the python library: TProcessPoolServer, which is a preforking python server.
> this server is sort of a mix between TForkingServer and TThreadPoolServer: It has a pool of workers that are preforked subprocesses.
> This approach allows the user to avoid the python GIL single processor limit in threading applications, and use a high performance server, and at the same time it avoids the need to fork a child process for each connection, as is the case with the forking server.
> I've benchmarked it to be about 5-6 times faster than TThreadPoolServer on a quad Corei7 CPU, and about the same amount faster than TForkingServer if you are not using persistent connections and forking a child for each request.
> The patch also updates the python unit tests to include tests for this server.
> Notes:
> 1. Of course this server has the limitations of forking regarding shared state and memory copying.
> 2. You should NOT kill the server with kill -9 - as this will not allow the parent process to terminate its children, resulting in orphaned processes that keep your socket open. either run stop() in your app or kill the process with SIGINT. It will respond to ctrl+C however
> 3. use setNumWorkers(n) before starting the server to determine how many processes you want to spawn. the default is 10.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (THRIFT-1083) Preforking python process pool server
Posted by "Dvir Volk (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dvir Volk updated THRIFT-1083:
------------------------------
Attachment: TProcessPoolServer.py
TProcessPoolServer.patch
The first file is a patch, the second is the original server module.
> Preforking python process pool server
> -------------------------------------
>
> Key: THRIFT-1083
> URL: https://issues.apache.org/jira/browse/THRIFT-1083
> Project: Thrift
> Issue Type: New Feature
> Components: Python - Library
> Affects Versions: 0.6
> Environment: linux. haven't tested on windows
> Reporter: Dvir Volk
> Priority: Minor
> Attachments: TProcessPoolServer.patch, TProcessPoolServer.py
>
>
> This patch adds a new server to the python library: TProcessPoolServer, which is a preforking python server.
> this server is sort of a mix between TForkingServer and TThreadPoolServer: It has a pool of workers that are preforked subprocesses.
> This approach allows the user to avoid the python GIL single processor limit in threading applications, and use a high performance server, and at the same time it avoids the need to fork a child process for each connection, as is the case with the forking server.
> I've benchmarked it to be about 5-6 times faster than TThreadPoolServer on a quad Corei7 CPU, and about the same amount faster than TForkingServer if you are not using persistent connections and forking a child for each request.
> The patch also updates the python unit tests to include tests for this server.
> Notes:
> 1. Of course this server has the limitations of forking regarding shared state and memory copying.
> 2. You should NOT kill the server with kill -9 - as this will not allow the parent process to terminate its children, resulting in orphaned processes that keep your socket open. either run stop() in your app or kill the process with SIGINT. It will respond to ctrl+C however
> 3. use setNumWorkers(n) before starting the server to determine how many processes you want to spawn. the default is 10.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (THRIFT-1083) Preforking python process pool
server
Posted by "Goir Riog (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250208#comment-13250208 ]
Goir Riog commented on THRIFT-1083:
-----------------------------------
This one should be made more public - took me a while to find it.
I do however have a problem with some shared Variables across the processes. Is there a way doing this with the current implementation ?
Adding them to the Process() method and passing it along in this class isn't a big problem. But they wont get passed along to the final function call in the generated Processor.process() method.
I could change the generated Code, but it would be nice to have it directly generated.
Is this worth a new Bug and change the generated code and the TProcessPoolServer class to accept like args or kwargs and pass them through ?
Thanks
> Preforking python process pool server
> -------------------------------------
>
> Key: THRIFT-1083
> URL: https://issues.apache.org/jira/browse/THRIFT-1083
> Project: Thrift
> Issue Type: New Feature
> Components: Python - Library
> Affects Versions: 0.6
> Environment: linux. haven't tested on windows
> Reporter: Dvir Volk
> Assignee: Dvir Volk
> Priority: Minor
> Fix For: 0.7
>
> Attachments: TProcessPoolServer.patch, TProcessPoolServer.py, TProcessPoolServer.py
>
>
> This patch adds a new server to the python library: TProcessPoolServer, which is a preforking python server.
> this server is sort of a mix between TForkingServer and TThreadPoolServer: It has a pool of workers that are preforked subprocesses.
> This approach allows the user to avoid the python GIL single processor limit in threading applications, and use a high performance server, and at the same time it avoids the need to fork a child process for each connection, as is the case with the forking server.
> I've benchmarked it to be about 5-6 times faster than TThreadPoolServer on a quad Corei7 CPU, and about the same amount faster than TForkingServer if you are not using persistent connections and forking a child for each request.
> The patch also updates the python unit tests to include tests for this server.
> Notes:
> 1. Of course this server has the limitations of forking regarding shared state and memory copying.
> 2. You should NOT kill the server with kill -9 - as this will not allow the parent process to terminate its children, resulting in orphaned processes that keep your socket open. either run stop() in your app or kill the process with SIGINT. It will respond to ctrl+C however
> 3. use setNumWorkers(n) before starting the server to determine how many processes you want to spawn. the default is 10.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (THRIFT-1083) Preforking python process pool
server
Posted by "Dvir Volk (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280789#comment-13280789 ]
Dvir Volk commented on THRIFT-1083:
-----------------------------------
@jonathan, let me know if it works for you, if so I'll clean the code up and suggest the newer version.
> Preforking python process pool server
> -------------------------------------
>
> Key: THRIFT-1083
> URL: https://issues.apache.org/jira/browse/THRIFT-1083
> Project: Thrift
> Issue Type: New Feature
> Components: Python - Library
> Affects Versions: 0.6
> Environment: linux. haven't tested on windows
> Reporter: Dvir Volk
> Assignee: Dvir Volk
> Priority: Minor
> Fix For: 0.7
>
> Attachments: TProcessPoolServer.patch, TProcessPoolServer.py, TProcessPoolServer.py
>
>
> This patch adds a new server to the python library: TProcessPoolServer, which is a preforking python server.
> this server is sort of a mix between TForkingServer and TThreadPoolServer: It has a pool of workers that are preforked subprocesses.
> This approach allows the user to avoid the python GIL single processor limit in threading applications, and use a high performance server, and at the same time it avoids the need to fork a child process for each connection, as is the case with the forking server.
> I've benchmarked it to be about 5-6 times faster than TThreadPoolServer on a quad Corei7 CPU, and about the same amount faster than TForkingServer if you are not using persistent connections and forking a child for each request.
> The patch also updates the python unit tests to include tests for this server.
> Notes:
> 1. Of course this server has the limitations of forking regarding shared state and memory copying.
> 2. You should NOT kill the server with kill -9 - as this will not allow the parent process to terminate its children, resulting in orphaned processes that keep your socket open. either run stop() in your app or kill the process with SIGINT. It will respond to ctrl+C however
> 3. use setNumWorkers(n) before starting the server to determine how many processes you want to spawn. the default is 10.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (THRIFT-1083) Preforking python process pool
server
Posted by "Jonathan Kaczynski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280321#comment-13280321 ]
Jonathan Kaczynski commented on THRIFT-1083:
--------------------------------------------
@dvir do you have any documentation on how to use the stop method in a service? I tried using the code from the test, http://svn.apache.org/repos/asf/thrift/trunk/test/py/TestServer.py, but the processes don't terminate.
> Preforking python process pool server
> -------------------------------------
>
> Key: THRIFT-1083
> URL: https://issues.apache.org/jira/browse/THRIFT-1083
> Project: Thrift
> Issue Type: New Feature
> Components: Python - Library
> Affects Versions: 0.6
> Environment: linux. haven't tested on windows
> Reporter: Dvir Volk
> Assignee: Dvir Volk
> Priority: Minor
> Fix For: 0.7
>
> Attachments: TProcessPoolServer.patch, TProcessPoolServer.py, TProcessPoolServer.py
>
>
> This patch adds a new server to the python library: TProcessPoolServer, which is a preforking python server.
> this server is sort of a mix between TForkingServer and TThreadPoolServer: It has a pool of workers that are preforked subprocesses.
> This approach allows the user to avoid the python GIL single processor limit in threading applications, and use a high performance server, and at the same time it avoids the need to fork a child process for each connection, as is the case with the forking server.
> I've benchmarked it to be about 5-6 times faster than TThreadPoolServer on a quad Corei7 CPU, and about the same amount faster than TForkingServer if you are not using persistent connections and forking a child for each request.
> The patch also updates the python unit tests to include tests for this server.
> Notes:
> 1. Of course this server has the limitations of forking regarding shared state and memory copying.
> 2. You should NOT kill the server with kill -9 - as this will not allow the parent process to terminate its children, resulting in orphaned processes that keep your socket open. either run stop() in your app or kill the process with SIGINT. It will respond to ctrl+C however
> 3. use setNumWorkers(n) before starting the server to determine how many processes you want to spawn. the default is 10.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira