You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by Matthew Rushton <mr...@yahoo.com> on 2009/08/17 18:39:53 UTC
Server Timeouts
Hi,
I'm trying to maintain a persistent connection to my erlang thrift server and things are timing out. The root of the cause can be found in thrift_buffered_transport.erl here:
read(Transport, Len) when is_integer(Len) ->
gen_server:call(Transport, {read, Len}, _Timeout=10000).
First I presume that the correct behavior of the server is to never timeout on the client? I did notice that the framed protocol does NOT contain the same timeout value. It also seems like the erlang server is hard coded to use the buffered transport (bottom of thrift_server.erl) but I guess that's a seperate issue. Basically I'm wondering the best way to fix this issue in the Erlang code, removing the timeout above would certainly do it or perhaps changing the looping behavior around it? Thanks.
-Matt
Re: Server Timeouts
Posted by Todd Lipcon <to...@cloudera.com>.
On Tue, Aug 18, 2009 at 12:48 AM, Matthew Rushton <mr...@yahoo.com>wrote:
> Thanks Todd, I ended up just using non-persistent connections as a
> temporary work around but I will get back to changing that when I have the
> chance. Out of curiosity were there any technical reasons why non-persistent
> connections were used over persistent ones at Facebook and Amie Street? I
> would expect performance to be significantly better with persistent
> connections.
> -Matt
>
I can't speak to Facebook, but I imagine there it's a case of there being
way too many frontend web servers for them all to keep persistent
connections open to all of the Erlang services. If I recall correctly from
some slides I read, the chat channel servers written in Erlang are
partitioned by userid, so any given request could need to go to any
particular group of Thrift servers, and the m*n connections would be too
much. Again, this is all surmised - perhaps Eugene or Chris can pipe up.
At AS it was mostly my distrust of PHP's persistent connection mechanisms.
If a PHP request were to die mid-response, the connection could be left in
an inconsistent state and cause all further requests on that connection to
fail as well. Given everything's on a local gigabit, the extra 0.4ms or so
of latency for the TCP handshake struck me as a fine price to pay for
increase stability, etc.
Hope that helps
-Todd
>
> --- On Mon, 8/17/09, Todd Lipcon <to...@cloudera.com> wrote:
>
> From: Todd Lipcon <to...@cloudera.com>
> Subject: Re: Server Timeouts
> To: thrift-dev@incubator.apache.org
> Date: Monday, August 17, 2009, 2:06 PM
>
> Hi Matt,
>
> I think there may be a JIRA for this. If you can't hunt it down, feel free
> to submit a patch.
>
> I think the original users of the thrift bindings (Amie Street and
> Facebook)
> both avoid persistent connections, so it never became an issue for us in
> production. The timeout should be configurable when the transport is
> initialized, IMO.
>
> As for the server, I seem to recall there being a {framed, true} option you
> could pass, but I may be thinking of the client.
>
> -Todd
>
> On Mon, Aug 17, 2009 at 9:39 AM, Matthew Rushton <mrushton7@yahoo.com
> >wrote:
>
> > Hi,
> > I'm trying to maintain a persistent connection to my erlang thrift
> server
> > and things are timing out. The root of the cause can be found in
> > thrift_buffered_transport.erl here:
> >
> > read(Transport, Len) when is_integer(Len) ->
> > gen_server:call(Transport, {read, Len}, _Timeout=10000).
> >
> > First I presume that the correct behavior of the server is to never
> timeout
> > on the client? I did notice that the framed protocol does NOT contain the
> > same timeout value. It also seems like the erlang server is hard coded to
> > use the buffered transport (bottom of thrift_server.erl) but I guess
> that's
> > a seperate issue. Basically I'm wondering the best way to fix this issue
> in
> > the Erlang code, removing the timeout above would certainly do it or
> perhaps
> > changing the looping behavior around it? Thanks.
> > -Matt
> >
> >
> >
> >
>
>
>
>
>
Re: Server Timeouts
Posted by David Reiss <dr...@facebook.com>.
The primary use case when we designed Thrift was a PHP client on a web server
connecting to a C++ server that implemented search. Persistent connections in
PHP connections are a bit tricky, and if we had every PHP process maintaining
a persistent connection to the search server, the amount of memory used by the
server would be very large. TCP setup cost on a LAN is very small compared to
the cost of performing one of our search queries.
--David
Matthew Rushton wrote:
> Thanks Todd, I ended up just using non-persistent connections as a temporary work around but I will get back to changing that when I have the chance. Out of curiosity were there any technical reasons why non-persistent connections were used over persistent ones at Facebook and Amie Street? I would expect performance to be significantly better with persistent connections.
> -Matt
>
> --- On Mon, 8/17/09, Todd Lipcon <to...@cloudera.com> wrote:
>
> From: Todd Lipcon <to...@cloudera.com>
> Subject: Re: Server Timeouts
> To: thrift-dev@incubator.apache.org
> Date: Monday, August 17, 2009, 2:06 PM
>
> Hi Matt,
>
> I think there may be a JIRA for this. If you can't hunt it down, feel free
> to submit a patch.
>
> I think the original users of the thrift bindings (Amie Street and Facebook)
> both avoid persistent connections, so it never became an issue for us in
> production. The timeout should be configurable when the transport is
> initialized, IMO.
>
> As for the server, I seem to recall there being a {framed, true} option you
> could pass, but I may be thinking of the client.
>
> -Todd
>
> On Mon, Aug 17, 2009 at 9:39 AM, Matthew Rushton <mr...@yahoo.com>wrote:
>
>> Hi,
>> I'm trying to maintain a persistent connection to my erlang thrift server
>> and things are timing out. The root of the cause can be found in
>> thrift_buffered_transport.erl here:
>>
>> read(Transport, Len) when is_integer(Len) ->
>> gen_server:call(Transport, {read, Len}, _Timeout=10000).
>>
>> First I presume that the correct behavior of the server is to never timeout
>> on the client? I did notice that the framed protocol does NOT contain the
>> same timeout value. It also seems like the erlang server is hard coded to
>> use the buffered transport (bottom of thrift_server.erl) but I guess that's
>> a seperate issue. Basically I'm wondering the best way to fix this issue in
>> the Erlang code, removing the timeout above would certainly do it or perhaps
>> changing the looping behavior around it? Thanks.
>> -Matt
>>
>>
>>
>>
>
>
>
>
Re: Server Timeouts
Posted by Matthew Rushton <mr...@yahoo.com>.
Thanks Todd, I ended up just using non-persistent connections as a temporary work around but I will get back to changing that when I have the chance. Out of curiosity were there any technical reasons why non-persistent connections were used over persistent ones at Facebook and Amie Street? I would expect performance to be significantly better with persistent connections.
-Matt
--- On Mon, 8/17/09, Todd Lipcon <to...@cloudera.com> wrote:
From: Todd Lipcon <to...@cloudera.com>
Subject: Re: Server Timeouts
To: thrift-dev@incubator.apache.org
Date: Monday, August 17, 2009, 2:06 PM
Hi Matt,
I think there may be a JIRA for this. If you can't hunt it down, feel free
to submit a patch.
I think the original users of the thrift bindings (Amie Street and Facebook)
both avoid persistent connections, so it never became an issue for us in
production. The timeout should be configurable when the transport is
initialized, IMO.
As for the server, I seem to recall there being a {framed, true} option you
could pass, but I may be thinking of the client.
-Todd
On Mon, Aug 17, 2009 at 9:39 AM, Matthew Rushton <mr...@yahoo.com>wrote:
> Hi,
> I'm trying to maintain a persistent connection to my erlang thrift server
> and things are timing out. The root of the cause can be found in
> thrift_buffered_transport.erl here:
>
> read(Transport, Len) when is_integer(Len) ->
> gen_server:call(Transport, {read, Len}, _Timeout=10000).
>
> First I presume that the correct behavior of the server is to never timeout
> on the client? I did notice that the framed protocol does NOT contain the
> same timeout value. It also seems like the erlang server is hard coded to
> use the buffered transport (bottom of thrift_server.erl) but I guess that's
> a seperate issue. Basically I'm wondering the best way to fix this issue in
> the Erlang code, removing the timeout above would certainly do it or perhaps
> changing the looping behavior around it? Thanks.
> -Matt
>
>
>
>
Re: Server Timeouts
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Matt,
I think there may be a JIRA for this. If you can't hunt it down, feel free
to submit a patch.
I think the original users of the thrift bindings (Amie Street and Facebook)
both avoid persistent connections, so it never became an issue for us in
production. The timeout should be configurable when the transport is
initialized, IMO.
As for the server, I seem to recall there being a {framed, true} option you
could pass, but I may be thinking of the client.
-Todd
On Mon, Aug 17, 2009 at 9:39 AM, Matthew Rushton <mr...@yahoo.com>wrote:
> Hi,
> I'm trying to maintain a persistent connection to my erlang thrift server
> and things are timing out. The root of the cause can be found in
> thrift_buffered_transport.erl here:
>
> read(Transport, Len) when is_integer(Len) ->
> gen_server:call(Transport, {read, Len}, _Timeout=10000).
>
> First I presume that the correct behavior of the server is to never timeout
> on the client? I did notice that the framed protocol does NOT contain the
> same timeout value. It also seems like the erlang server is hard coded to
> use the buffered transport (bottom of thrift_server.erl) but I guess that's
> a seperate issue. Basically I'm wondering the best way to fix this issue in
> the Erlang code, removing the timeout above would certainly do it or perhaps
> changing the looping behavior around it? Thanks.
> -Matt
>
>
>
>