You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@thrift.apache.org by Mark Slee <ms...@facebook.com> on 2010/08/02 21:57:42 UTC
RE: Short question re. threading with a Python server
Generally, no. Python's threading.local is a specific way of offering thread-local storage.
You should NOT use threading.local if you're trying to safely share data between threads. That's not what it does -- rather it actually exposes a *different* copy of the data to each thread.
So, you should use it if you actually want N copies of some data (where N is the number of threads), but you want a simple interface to refer to them all by the same name (with the clear understanding that you are actually accessing N different pieces of data).
There are certainly situations in which this can be useful in a threaded Thrift server, but it's by no means a general rule.
-----Original Message-----
From: Phillip B Oldham [mailto:phillip.oldham@gmail.com]
Sent: Friday, July 30, 2010 2:39 AM
To: thrift-user
Subject: Short question re. threading with a Python server
Should Python classes that are run via the
TThreaded/TThreadPool/TNonBlocking server inherit from
`threading.local`, in order to be thread safe?
--
Phillip B Oldham
phillip.oldham@gmail.com
+44 (0) 7525 01 09 01
RE: Short question re. threading with a Python server
Posted by Mark Slee <ms...@facebook.com>.
Gotcha. Slightly weird that threading.local would fix that. It sounds like what you may have potentially just done is create N MySQL clients, one-per each thread (resulting in each thread just blocking on its own query, rather than each thread blocking on getting access to that same MySQL connection when someone else is using it).
Out of curiosity, does "causing error responses for other clients" mean causing timeouts? Or was there actually some kind of error being generated from the internals?
-----Original Message-----
From: Phillip B Oldham [mailto:phillip.oldham@gmail.com]
Sent: Friday, August 06, 2010 6:08 AM
To: thrift-user@incubator.apache.org
Subject: Re: Short question re. threading with a Python server
Seems my error was due to threading, specifically with the MySQL
client. I had to alter the handler class so that it extended
`threading.local` otherwise a MySQL query would block the whole
process, causing error responses for other clients.
On Tue, Aug 3, 2010 at 2:41 PM, Phillip B Oldham
<ph...@gmail.com> wrote:
> Thanks Mark, that's really helpful. We're seeing a very odd bottleneck
> and thought it could've been down to issues with threading in our
> Python server. We've no shared state except what is persisted through
> memcached and MySQL, so it mustn't be that. I'll continue to dig
> around.
>
> Cheers!
>
> On Mon, Aug 2, 2010 at 8:57 PM, Mark Slee <ms...@facebook.com> wrote:
>> Generally, no. Python's threading.local is a specific way of offering thread-local storage.
>>
>> You should NOT use threading.local if you're trying to safely share data between threads. That's not what it does -- rather it actually exposes a *different* copy of the data to each thread.
>>
>> So, you should use it if you actually want N copies of some data (where N is the number of threads), but you want a simple interface to refer to them all by the same name (with the clear understanding that you are actually accessing N different pieces of data).
>>
>> There are certainly situations in which this can be useful in a threaded Thrift server, but it's by no means a general rule.
>>
>> -----Original Message-----
>> From: Phillip B Oldham [mailto:phillip.oldham@gmail.com]
>> Sent: Friday, July 30, 2010 2:39 AM
>> To: thrift-user
>> Subject: Short question re. threading with a Python server
>>
>> Should Python classes that are run via the
>> TThreaded/TThreadPool/TNonBlocking server inherit from
>> `threading.local`, in order to be thread safe?
>>
>> --
>> Phillip B Oldham
>> phillip.oldham@gmail.com
>> +44 (0) 7525 01 09 01
>>
>
>
>
> --
> Phillip B Oldham
> phillip.oldham@gmail.com
> +44 (0) 7525 01 09 01
>
--
Phillip B Oldham
phillip.oldham@gmail.com
+44 (0) 7525 01 09 01
Re: Short question re. threading with a Python server
Posted by Phillip B Oldham <ph...@gmail.com>.
Seems my error was due to threading, specifically with the MySQL
client. I had to alter the handler class so that it extended
`threading.local` otherwise a MySQL query would block the whole
process, causing error responses for other clients.
On Tue, Aug 3, 2010 at 2:41 PM, Phillip B Oldham
<ph...@gmail.com> wrote:
> Thanks Mark, that's really helpful. We're seeing a very odd bottleneck
> and thought it could've been down to issues with threading in our
> Python server. We've no shared state except what is persisted through
> memcached and MySQL, so it mustn't be that. I'll continue to dig
> around.
>
> Cheers!
>
> On Mon, Aug 2, 2010 at 8:57 PM, Mark Slee <ms...@facebook.com> wrote:
>> Generally, no. Python's threading.local is a specific way of offering thread-local storage.
>>
>> You should NOT use threading.local if you're trying to safely share data between threads. That's not what it does -- rather it actually exposes a *different* copy of the data to each thread.
>>
>> So, you should use it if you actually want N copies of some data (where N is the number of threads), but you want a simple interface to refer to them all by the same name (with the clear understanding that you are actually accessing N different pieces of data).
>>
>> There are certainly situations in which this can be useful in a threaded Thrift server, but it's by no means a general rule.
>>
>> -----Original Message-----
>> From: Phillip B Oldham [mailto:phillip.oldham@gmail.com]
>> Sent: Friday, July 30, 2010 2:39 AM
>> To: thrift-user
>> Subject: Short question re. threading with a Python server
>>
>> Should Python classes that are run via the
>> TThreaded/TThreadPool/TNonBlocking server inherit from
>> `threading.local`, in order to be thread safe?
>>
>> --
>> Phillip B Oldham
>> phillip.oldham@gmail.com
>> +44 (0) 7525 01 09 01
>>
>
>
>
> --
> Phillip B Oldham
> phillip.oldham@gmail.com
> +44 (0) 7525 01 09 01
>
--
Phillip B Oldham
phillip.oldham@gmail.com
+44 (0) 7525 01 09 01
Re: Short question re. threading with a Python server
Posted by Phillip B Oldham <ph...@gmail.com>.
Thanks Mark, that's really helpful. We're seeing a very odd bottleneck
and thought it could've been down to issues with threading in our
Python server. We've no shared state except what is persisted through
memcached and MySQL, so it mustn't be that. I'll continue to dig
around.
Cheers!
On Mon, Aug 2, 2010 at 8:57 PM, Mark Slee <ms...@facebook.com> wrote:
> Generally, no. Python's threading.local is a specific way of offering thread-local storage.
>
> You should NOT use threading.local if you're trying to safely share data between threads. That's not what it does -- rather it actually exposes a *different* copy of the data to each thread.
>
> So, you should use it if you actually want N copies of some data (where N is the number of threads), but you want a simple interface to refer to them all by the same name (with the clear understanding that you are actually accessing N different pieces of data).
>
> There are certainly situations in which this can be useful in a threaded Thrift server, but it's by no means a general rule.
>
> -----Original Message-----
> From: Phillip B Oldham [mailto:phillip.oldham@gmail.com]
> Sent: Friday, July 30, 2010 2:39 AM
> To: thrift-user
> Subject: Short question re. threading with a Python server
>
> Should Python classes that are run via the
> TThreaded/TThreadPool/TNonBlocking server inherit from
> `threading.local`, in order to be thread safe?
>
> --
> Phillip B Oldham
> phillip.oldham@gmail.com
> +44 (0) 7525 01 09 01
>
--
Phillip B Oldham
phillip.oldham@gmail.com
+44 (0) 7525 01 09 01