You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@thrift.apache.org by Utku Can Topçu <ut...@topcu.gen.tr> on 2010/01/05 09:02:09 UTC
PHP Client fails to read correct return values from the Java Service
Dear Thrift users and Developers,
I've written a simple service which has the following thrift definition:
service CounterService {
string increment(1:i64 itemId),
string getCount(1:i64 itemId),
}
In our system, we're using PHP as the client and Java as the Server,
Following the sample client and server stub examples included in the
tutorial package, I haven't made any big differences in both the Client and
Server code (and of course never touched the auto-generated code);
The Java Server makes use of the following simple stub:
--Server.java--
CounterHandler handler = new CounterHandler();
CounterService.Processor processor = new CounterService.Processor(handler);
TServerTransport serverTransport;
serverTransport = new TServerSocket(8888);
server = new TThreadPoolServer(processor, serverTransport);
---
The CounterHandler implements CounterService.Iface
In the PHP Client code I have,
--Client.php--
$server = '192.168.1.64';
$socket = new TSocket($server, 8888, TRUE); //Using persistent connection to
the Java Service
$transport = new TBufferedTransport($socket, 1024, 1024);
$protocol = new TBinaryProtocol($transport);
$client = new CounterServiceClient($protocol);
$transport->open();
$jcount = $client->increment($auctionId);
$transport->close();
---
The problem is the fact that; under our production load (1500 persistent
socket connections from 7 PHP clients, about a total of 300 requests per
second), the PHP Client fails to get the correct increment(id) value from
the Buffer, it somehow gets previous replies from the Server.
I'm logging the Java Server transactions, the Java Server seems to work
fine; I'm suspecting that the problem occurs because of the PHP client
libraries.
We are running PHP in lighttpd and fastcgi btw.
Any comments on solving the problem is welcome.
Regards,
Utku
Re: PHP Client fails to read correct return values from the Java
Service
Posted by Utku Can Topçu <ut...@topcu.gen.tr>.
Hello Mark,
I tried disabling the persistent connections and it worked fine without any
problem, however this time I've faced a serious connection overhead in a few
moments.
Here's the sample server code; I've just taken it from the tutorial.
try {
CounterHandler handler = new CounterHandler();
CounterService.Processor processor = new CounterService.Processor(handler);
TServerTransport serverTransport;
serverTransport = new TServerSocket(8888);
TServer server = new TThreadPoolServer(processor, serverTransport);
logger.info("Starting the server...");
server.serve();
} catch (TTransportException e) {
}
Where CounterHandler implements CounterService.Iface
public class CounterHandler implements CounterService.Iface {
.
.
public String increment(long auctionId) throws TException {
CounterWorker worker = CounterWorker.getInstance(); //Singleton
Worker
long count = worker.increment(auctionId, 1); //increment the
counter of the id by one
logger.info("increment("+auctionId + "):" + count + "\t[csize:"
+ worker.counterSize() +
",isize:"+worker.incrementSize()+"][call:" + callCount + "]");
return String.valueOf(count);
}
...
what basically CounterWorker's "long increment(long id, long increment)"
method does is, execute two simple update and select queries on MySQL by
making use of connection pooling.
The problem is the fact that, I'm just calling the increment method from the
PHP Client, and somehow the returning result belongs to a different id from
the provided one.
I fear the problem occurs after several timeouts on client side.
Thanks for your helps and comments,
Utku
On Fri, Jan 15, 2010 at 7:12 AM, Todd Lipcon <to...@cloudera.com> wrote:
> On Thu, Jan 14, 2010 at 6:36 PM, Mark Slee <ms...@facebook.com> wrote:
>
> > Would need more info to debug this. Do you mind sharing more of your
> server
> > implementation code? The client code looks fine to me. I'm not really
> clear
> > on how you'd get the result of an old call when you're using a new socket
> > each time, unless there is a bug with PHP's underlying persistent socket
> > implementation leaving old data in the socket buffer (I've never seen
> this
> > before, and it seems like it would create major problems if this could
> ever
> > happen -- I would have assumed that pfsockopen() would always reset the
> recv
> > buffer).
> >
> >
> Isn't it impossible/racey for PHP to guarantee this? That is to say, it can
> *think* the buffer's clear, then hand off the socket to the client, and
> then
> the packets arrive.
>
> -Todd
>
>
> > Can you try running this without using PHP persistent sockets? See if it
> > fixes the issue to open/close a new socket every time?
> >
> > -----Original Message-----
> > From: Utku Can Topçu [mailto:utku@topcu.gen.tr]
> > Sent: Tuesday, January 05, 2010 12:02 AM
> > To: thrift-user@incubator.apache.org
> > Subject: PHP Client fails to read correct return values from the Java
> > Service
> >
> > Dear Thrift users and Developers,
> >
> > I've written a simple service which has the following thrift definition:
> >
> > service CounterService {
> > string increment(1:i64 itemId),
> > string getCount(1:i64 itemId),
> > }
> >
> > In our system, we're using PHP as the client and Java as the Server,
> > Following the sample client and server stub examples included in the
> > tutorial package, I haven't made any big differences in both the Client
> and
> > Server code (and of course never touched the auto-generated code);
> >
> > The Java Server makes use of the following simple stub:
> >
> > --Server.java--
> > CounterHandler handler = new CounterHandler();
> > CounterService.Processor processor = new
> CounterService.Processor(handler);
> > TServerTransport serverTransport;
> > serverTransport = new TServerSocket(8888);
> > server = new TThreadPoolServer(processor, serverTransport);
> > ---
> >
> > The CounterHandler implements CounterService.Iface
> >
> > In the PHP Client code I have,
> >
> > --Client.php--
> > $server = '192.168.1.64';
> > $socket = new TSocket($server, 8888, TRUE); //Using persistent connection
> > to
> > the Java Service
> > $transport = new TBufferedTransport($socket, 1024, 1024);
> > $protocol = new TBinaryProtocol($transport);
> > $client = new CounterServiceClient($protocol);
> > $transport->open();
> > $jcount = $client->increment($auctionId);
> > $transport->close();
> > ---
> >
> > The problem is the fact that; under our production load (1500 persistent
> > socket connections from 7 PHP clients, about a total of 300 requests per
> > second), the PHP Client fails to get the correct increment(id) value from
> > the Buffer, it somehow gets previous replies from the Server.
> >
> > I'm logging the Java Server transactions, the Java Server seems to work
> > fine; I'm suspecting that the problem occurs because of the PHP client
> > libraries.
> >
> > We are running PHP in lighttpd and fastcgi btw.
> >
> > Any comments on solving the problem is welcome.
> >
> > Regards,
> > Utku
> >
>
Re: PHP Client fails to read correct return values from the Java
Service
Posted by Utku Can Topçu <ut...@topcu.gen.tr>.
Hello David,
I think that would certainly solve the problem, however I couldn't manage to
throw the persistent connection away and failed to create a new one. Have
you got any guideline or code-samples for handling this issue?
Best Regards,
Utku
On Tue, Feb 9, 2010 at 6:15 PM, David Reiss <dr...@facebook.com> wrote:
> After a read timeout, you have to throw the connection away and create a
> new one.
>
> Utku Can Topçu wrote:
> > Hello Again,
> >
> > I'm re-volting the issue,
> >
> > I've conducted several experiments and the problem does seem to continue.
> >
> > First of all I tried using the TFramedTransport, at first it worked fine
> but
> > this only lasted until several client timeouts occurred.
> >
> > Let me explain the problem again,
> >
> > Say, there are n consecutive php requests on the web server working on
> the
> > same php-process, calling the service for the for a simple function like
> > Item getItem(1:i64 requestedItemId)
> > the getItem function connects to the DB and fetches and forms an object
> for
> > generating the Item requested.
> > The contract for the Item object result is, Item.id = requestedItemId
> >
> > If I'm using a persistent connection, all requests running under the same
> > php-process defined above use the same socket and buffer for service
> calls.
> >
> > Now,
> > at time t1 php client calls getItem(1:i64 id1) with the timeout 1000 ms.
> > at time t1+1000 php client times out and exits
> > at time t2 (t2>t1+1000) Java service returns the Item
> > at time t3 (t3 is just after t2) php client calls getItem(1:i64 id2) with
> > the timeout 1000 ms.
> > at time t4 (t4 is just after t3) php client reads the Item instance which
> > was the response for the call at "time t1", so what I get is Item.id =
> id1
> > whereas it should have beed, Item.id = id2
> >
> > If I'm not using persistent connections, such problem never happens, but
> > this time I seriously face a connection overhead which fails eventually.
> >
> > Can you suggest me scheme for working around this?
> >
> > Best Regards,
> > Utku
> >
> >
> > On Sat, Jan 16, 2010 at 1:17 AM, Mark Slee <ms...@facebook.com> wrote:
> >
> >> Yes, you're right. That's definitely possible, and it sounds like that's
> >> probably what is happening here.
> >>
> >> I was referring to was the fact that persistent sockets are never shared
> >> across separate apache processes (in the OS sense), so the pfsockopen()
> call
> >> should only return a socket that's done being used by the last page
> request.
> >>
> >> I was wrongly assuming the last page request properly completed use of
> the
> >> socket. If the client side experiences a read timeout, it'll close up
> making
> >> it possible for a subsequent request to get that same socket later, and
> then
> >> the server response will come back.
> >>
> >> The right fix for this is to build proper sequence-id validation into
> the
> >> PHP client code. We actually do send a unique sequence id with each RPC
> >> request that should be echoed back by the server, but currently we're
> not
> >> tracking state in the client to match these.
> >>
> >> Cheers,
> >> Mark
> >>
> >> -----Original Message-----
> >> From: Todd Lipcon [mailto:todd@cloudera.com]
> >> Sent: Thursday, January 14, 2010 9:12 PM
> >> To: thrift-user@incubator.apache.org
> >> Subject: Re: PHP Client fails to read correct return values from the
> Java
> >> Service
> >>
> >> On Thu, Jan 14, 2010 at 6:36 PM, Mark Slee <ms...@facebook.com> wrote:
> >>
> >>> Would need more info to debug this. Do you mind sharing more of your
> >> server
> >>> implementation code? The client code looks fine to me. I'm not really
> >> clear
> >>> on how you'd get the result of an old call when you're using a new
> socket
> >>> each time, unless there is a bug with PHP's underlying persistent
> socket
> >>> implementation leaving old data in the socket buffer (I've never seen
> >> this
> >>> before, and it seems like it would create major problems if this could
> >> ever
> >>> happen -- I would have assumed that pfsockopen() would always reset the
> >> recv
> >>> buffer).
> >>>
> >>>
> >> Isn't it impossible/racey for PHP to guarantee this? That is to say, it
> can
> >> *think* the buffer's clear, then hand off the socket to the client, and
> >> then
> >> the packets arrive.
> >>
> >> -Todd
> >>
> >>
> >>> Can you try running this without using PHP persistent sockets? See if
> it
> >>> fixes the issue to open/close a new socket every time?
> >>>
> >>> -----Original Message-----
> >>> From: Utku Can Topçu [mailto:utku@topcu.gen.tr]
> >>> Sent: Tuesday, January 05, 2010 12:02 AM
> >>> To: thrift-user@incubator.apache.org
> >>> Subject: PHP Client fails to read correct return values from the Java
> >>> Service
> >>>
> >>> Dear Thrift users and Developers,
> >>>
> >>> I've written a simple service which has the following thrift
> definition:
> >>>
> >>> service CounterService {
> >>> string increment(1:i64 itemId),
> >>> string getCount(1:i64 itemId),
> >>> }
> >>>
> >>> In our system, we're using PHP as the client and Java as the Server,
> >>> Following the sample client and server stub examples included in the
> >>> tutorial package, I haven't made any big differences in both the Client
> >> and
> >>> Server code (and of course never touched the auto-generated code);
> >>>
> >>> The Java Server makes use of the following simple stub:
> >>>
> >>> --Server.java--
> >>> CounterHandler handler = new CounterHandler();
> >>> CounterService.Processor processor = new
> >> CounterService.Processor(handler);
> >>> TServerTransport serverTransport;
> >>> serverTransport = new TServerSocket(8888);
> >>> server = new TThreadPoolServer(processor, serverTransport);
> >>> ---
> >>>
> >>> The CounterHandler implements CounterService.Iface
> >>>
> >>> In the PHP Client code I have,
> >>>
> >>> --Client.php--
> >>> $server = '192.168.1.64';
> >>> $socket = new TSocket($server, 8888, TRUE); //Using persistent
> connection
> >>> to
> >>> the Java Service
> >>> $transport = new TBufferedTransport($socket, 1024, 1024);
> >>> $protocol = new TBinaryProtocol($transport);
> >>> $client = new CounterServiceClient($protocol);
> >>> $transport->open();
> >>> $jcount = $client->increment($auctionId);
> >>> $transport->close();
> >>> ---
> >>>
> >>> The problem is the fact that; under our production load (1500
> persistent
> >>> socket connections from 7 PHP clients, about a total of 300 requests
> per
> >>> second), the PHP Client fails to get the correct increment(id) value
> from
> >>> the Buffer, it somehow gets previous replies from the Server.
> >>>
> >>> I'm logging the Java Server transactions, the Java Server seems to work
> >>> fine; I'm suspecting that the problem occurs because of the PHP client
> >>> libraries.
> >>>
> >>> We are running PHP in lighttpd and fastcgi btw.
> >>>
> >>> Any comments on solving the problem is welcome.
> >>>
> >>> Regards,
> >>> Utku
> >>>
>
Re: PHP Client fails to read correct return values from the Java
Service
Posted by David Reiss <dr...@facebook.com>.
After a read timeout, you have to throw the connection away and create a new one.
Utku Can Topçu wrote:
> Hello Again,
>
> I'm re-volting the issue,
>
> I've conducted several experiments and the problem does seem to continue.
>
> First of all I tried using the TFramedTransport, at first it worked fine but
> this only lasted until several client timeouts occurred.
>
> Let me explain the problem again,
>
> Say, there are n consecutive php requests on the web server working on the
> same php-process, calling the service for the for a simple function like
> Item getItem(1:i64 requestedItemId)
> the getItem function connects to the DB and fetches and forms an object for
> generating the Item requested.
> The contract for the Item object result is, Item.id = requestedItemId
>
> If I'm using a persistent connection, all requests running under the same
> php-process defined above use the same socket and buffer for service calls.
>
> Now,
> at time t1 php client calls getItem(1:i64 id1) with the timeout 1000 ms.
> at time t1+1000 php client times out and exits
> at time t2 (t2>t1+1000) Java service returns the Item
> at time t3 (t3 is just after t2) php client calls getItem(1:i64 id2) with
> the timeout 1000 ms.
> at time t4 (t4 is just after t3) php client reads the Item instance which
> was the response for the call at "time t1", so what I get is Item.id = id1
> whereas it should have beed, Item.id = id2
>
> If I'm not using persistent connections, such problem never happens, but
> this time I seriously face a connection overhead which fails eventually.
>
> Can you suggest me scheme for working around this?
>
> Best Regards,
> Utku
>
>
> On Sat, Jan 16, 2010 at 1:17 AM, Mark Slee <ms...@facebook.com> wrote:
>
>> Yes, you're right. That's definitely possible, and it sounds like that's
>> probably what is happening here.
>>
>> I was referring to was the fact that persistent sockets are never shared
>> across separate apache processes (in the OS sense), so the pfsockopen() call
>> should only return a socket that's done being used by the last page request.
>>
>> I was wrongly assuming the last page request properly completed use of the
>> socket. If the client side experiences a read timeout, it'll close up making
>> it possible for a subsequent request to get that same socket later, and then
>> the server response will come back.
>>
>> The right fix for this is to build proper sequence-id validation into the
>> PHP client code. We actually do send a unique sequence id with each RPC
>> request that should be echoed back by the server, but currently we're not
>> tracking state in the client to match these.
>>
>> Cheers,
>> Mark
>>
>> -----Original Message-----
>> From: Todd Lipcon [mailto:todd@cloudera.com]
>> Sent: Thursday, January 14, 2010 9:12 PM
>> To: thrift-user@incubator.apache.org
>> Subject: Re: PHP Client fails to read correct return values from the Java
>> Service
>>
>> On Thu, Jan 14, 2010 at 6:36 PM, Mark Slee <ms...@facebook.com> wrote:
>>
>>> Would need more info to debug this. Do you mind sharing more of your
>> server
>>> implementation code? The client code looks fine to me. I'm not really
>> clear
>>> on how you'd get the result of an old call when you're using a new socket
>>> each time, unless there is a bug with PHP's underlying persistent socket
>>> implementation leaving old data in the socket buffer (I've never seen
>> this
>>> before, and it seems like it would create major problems if this could
>> ever
>>> happen -- I would have assumed that pfsockopen() would always reset the
>> recv
>>> buffer).
>>>
>>>
>> Isn't it impossible/racey for PHP to guarantee this? That is to say, it can
>> *think* the buffer's clear, then hand off the socket to the client, and
>> then
>> the packets arrive.
>>
>> -Todd
>>
>>
>>> Can you try running this without using PHP persistent sockets? See if it
>>> fixes the issue to open/close a new socket every time?
>>>
>>> -----Original Message-----
>>> From: Utku Can Topçu [mailto:utku@topcu.gen.tr]
>>> Sent: Tuesday, January 05, 2010 12:02 AM
>>> To: thrift-user@incubator.apache.org
>>> Subject: PHP Client fails to read correct return values from the Java
>>> Service
>>>
>>> Dear Thrift users and Developers,
>>>
>>> I've written a simple service which has the following thrift definition:
>>>
>>> service CounterService {
>>> string increment(1:i64 itemId),
>>> string getCount(1:i64 itemId),
>>> }
>>>
>>> In our system, we're using PHP as the client and Java as the Server,
>>> Following the sample client and server stub examples included in the
>>> tutorial package, I haven't made any big differences in both the Client
>> and
>>> Server code (and of course never touched the auto-generated code);
>>>
>>> The Java Server makes use of the following simple stub:
>>>
>>> --Server.java--
>>> CounterHandler handler = new CounterHandler();
>>> CounterService.Processor processor = new
>> CounterService.Processor(handler);
>>> TServerTransport serverTransport;
>>> serverTransport = new TServerSocket(8888);
>>> server = new TThreadPoolServer(processor, serverTransport);
>>> ---
>>>
>>> The CounterHandler implements CounterService.Iface
>>>
>>> In the PHP Client code I have,
>>>
>>> --Client.php--
>>> $server = '192.168.1.64';
>>> $socket = new TSocket($server, 8888, TRUE); //Using persistent connection
>>> to
>>> the Java Service
>>> $transport = new TBufferedTransport($socket, 1024, 1024);
>>> $protocol = new TBinaryProtocol($transport);
>>> $client = new CounterServiceClient($protocol);
>>> $transport->open();
>>> $jcount = $client->increment($auctionId);
>>> $transport->close();
>>> ---
>>>
>>> The problem is the fact that; under our production load (1500 persistent
>>> socket connections from 7 PHP clients, about a total of 300 requests per
>>> second), the PHP Client fails to get the correct increment(id) value from
>>> the Buffer, it somehow gets previous replies from the Server.
>>>
>>> I'm logging the Java Server transactions, the Java Server seems to work
>>> fine; I'm suspecting that the problem occurs because of the PHP client
>>> libraries.
>>>
>>> We are running PHP in lighttpd and fastcgi btw.
>>>
>>> Any comments on solving the problem is welcome.
>>>
>>> Regards,
>>> Utku
>>>
Re: PHP Client fails to read correct return values from the Java
Service
Posted by Utku Can Topçu <ut...@topcu.gen.tr>.
Hello Again,
I'm re-volting the issue,
I've conducted several experiments and the problem does seem to continue.
First of all I tried using the TFramedTransport, at first it worked fine but
this only lasted until several client timeouts occurred.
Let me explain the problem again,
Say, there are n consecutive php requests on the web server working on the
same php-process, calling the service for the for a simple function like
Item getItem(1:i64 requestedItemId)
the getItem function connects to the DB and fetches and forms an object for
generating the Item requested.
The contract for the Item object result is, Item.id = requestedItemId
If I'm using a persistent connection, all requests running under the same
php-process defined above use the same socket and buffer for service calls.
Now,
at time t1 php client calls getItem(1:i64 id1) with the timeout 1000 ms.
at time t1+1000 php client times out and exits
at time t2 (t2>t1+1000) Java service returns the Item
at time t3 (t3 is just after t2) php client calls getItem(1:i64 id2) with
the timeout 1000 ms.
at time t4 (t4 is just after t3) php client reads the Item instance which
was the response for the call at "time t1", so what I get is Item.id = id1
whereas it should have beed, Item.id = id2
If I'm not using persistent connections, such problem never happens, but
this time I seriously face a connection overhead which fails eventually.
Can you suggest me scheme for working around this?
Best Regards,
Utku
On Sat, Jan 16, 2010 at 1:17 AM, Mark Slee <ms...@facebook.com> wrote:
> Yes, you're right. That's definitely possible, and it sounds like that's
> probably what is happening here.
>
> I was referring to was the fact that persistent sockets are never shared
> across separate apache processes (in the OS sense), so the pfsockopen() call
> should only return a socket that's done being used by the last page request.
>
> I was wrongly assuming the last page request properly completed use of the
> socket. If the client side experiences a read timeout, it'll close up making
> it possible for a subsequent request to get that same socket later, and then
> the server response will come back.
>
> The right fix for this is to build proper sequence-id validation into the
> PHP client code. We actually do send a unique sequence id with each RPC
> request that should be echoed back by the server, but currently we're not
> tracking state in the client to match these.
>
> Cheers,
> Mark
>
> -----Original Message-----
> From: Todd Lipcon [mailto:todd@cloudera.com]
> Sent: Thursday, January 14, 2010 9:12 PM
> To: thrift-user@incubator.apache.org
> Subject: Re: PHP Client fails to read correct return values from the Java
> Service
>
> On Thu, Jan 14, 2010 at 6:36 PM, Mark Slee <ms...@facebook.com> wrote:
>
> > Would need more info to debug this. Do you mind sharing more of your
> server
> > implementation code? The client code looks fine to me. I'm not really
> clear
> > on how you'd get the result of an old call when you're using a new socket
> > each time, unless there is a bug with PHP's underlying persistent socket
> > implementation leaving old data in the socket buffer (I've never seen
> this
> > before, and it seems like it would create major problems if this could
> ever
> > happen -- I would have assumed that pfsockopen() would always reset the
> recv
> > buffer).
> >
> >
> Isn't it impossible/racey for PHP to guarantee this? That is to say, it can
> *think* the buffer's clear, then hand off the socket to the client, and
> then
> the packets arrive.
>
> -Todd
>
>
> > Can you try running this without using PHP persistent sockets? See if it
> > fixes the issue to open/close a new socket every time?
> >
> > -----Original Message-----
> > From: Utku Can Topçu [mailto:utku@topcu.gen.tr]
> > Sent: Tuesday, January 05, 2010 12:02 AM
> > To: thrift-user@incubator.apache.org
> > Subject: PHP Client fails to read correct return values from the Java
> > Service
> >
> > Dear Thrift users and Developers,
> >
> > I've written a simple service which has the following thrift definition:
> >
> > service CounterService {
> > string increment(1:i64 itemId),
> > string getCount(1:i64 itemId),
> > }
> >
> > In our system, we're using PHP as the client and Java as the Server,
> > Following the sample client and server stub examples included in the
> > tutorial package, I haven't made any big differences in both the Client
> and
> > Server code (and of course never touched the auto-generated code);
> >
> > The Java Server makes use of the following simple stub:
> >
> > --Server.java--
> > CounterHandler handler = new CounterHandler();
> > CounterService.Processor processor = new
> CounterService.Processor(handler);
> > TServerTransport serverTransport;
> > serverTransport = new TServerSocket(8888);
> > server = new TThreadPoolServer(processor, serverTransport);
> > ---
> >
> > The CounterHandler implements CounterService.Iface
> >
> > In the PHP Client code I have,
> >
> > --Client.php--
> > $server = '192.168.1.64';
> > $socket = new TSocket($server, 8888, TRUE); //Using persistent connection
> > to
> > the Java Service
> > $transport = new TBufferedTransport($socket, 1024, 1024);
> > $protocol = new TBinaryProtocol($transport);
> > $client = new CounterServiceClient($protocol);
> > $transport->open();
> > $jcount = $client->increment($auctionId);
> > $transport->close();
> > ---
> >
> > The problem is the fact that; under our production load (1500 persistent
> > socket connections from 7 PHP clients, about a total of 300 requests per
> > second), the PHP Client fails to get the correct increment(id) value from
> > the Buffer, it somehow gets previous replies from the Server.
> >
> > I'm logging the Java Server transactions, the Java Server seems to work
> > fine; I'm suspecting that the problem occurs because of the PHP client
> > libraries.
> >
> > We are running PHP in lighttpd and fastcgi btw.
> >
> > Any comments on solving the problem is welcome.
> >
> > Regards,
> > Utku
> >
>
RE: PHP Client fails to read correct return values from the Java
Service
Posted by Mark Slee <ms...@facebook.com>.
Yes, you're right. That's definitely possible, and it sounds like that's probably what is happening here.
I was referring to was the fact that persistent sockets are never shared across separate apache processes (in the OS sense), so the pfsockopen() call should only return a socket that's done being used by the last page request.
I was wrongly assuming the last page request properly completed use of the socket. If the client side experiences a read timeout, it'll close up making it possible for a subsequent request to get that same socket later, and then the server response will come back.
The right fix for this is to build proper sequence-id validation into the PHP client code. We actually do send a unique sequence id with each RPC request that should be echoed back by the server, but currently we're not tracking state in the client to match these.
Cheers,
Mark
-----Original Message-----
From: Todd Lipcon [mailto:todd@cloudera.com]
Sent: Thursday, January 14, 2010 9:12 PM
To: thrift-user@incubator.apache.org
Subject: Re: PHP Client fails to read correct return values from the Java Service
On Thu, Jan 14, 2010 at 6:36 PM, Mark Slee <ms...@facebook.com> wrote:
> Would need more info to debug this. Do you mind sharing more of your server
> implementation code? The client code looks fine to me. I'm not really clear
> on how you'd get the result of an old call when you're using a new socket
> each time, unless there is a bug with PHP's underlying persistent socket
> implementation leaving old data in the socket buffer (I've never seen this
> before, and it seems like it would create major problems if this could ever
> happen -- I would have assumed that pfsockopen() would always reset the recv
> buffer).
>
>
Isn't it impossible/racey for PHP to guarantee this? That is to say, it can
*think* the buffer's clear, then hand off the socket to the client, and then
the packets arrive.
-Todd
> Can you try running this without using PHP persistent sockets? See if it
> fixes the issue to open/close a new socket every time?
>
> -----Original Message-----
> From: Utku Can Topçu [mailto:utku@topcu.gen.tr]
> Sent: Tuesday, January 05, 2010 12:02 AM
> To: thrift-user@incubator.apache.org
> Subject: PHP Client fails to read correct return values from the Java
> Service
>
> Dear Thrift users and Developers,
>
> I've written a simple service which has the following thrift definition:
>
> service CounterService {
> string increment(1:i64 itemId),
> string getCount(1:i64 itemId),
> }
>
> In our system, we're using PHP as the client and Java as the Server,
> Following the sample client and server stub examples included in the
> tutorial package, I haven't made any big differences in both the Client and
> Server code (and of course never touched the auto-generated code);
>
> The Java Server makes use of the following simple stub:
>
> --Server.java--
> CounterHandler handler = new CounterHandler();
> CounterService.Processor processor = new CounterService.Processor(handler);
> TServerTransport serverTransport;
> serverTransport = new TServerSocket(8888);
> server = new TThreadPoolServer(processor, serverTransport);
> ---
>
> The CounterHandler implements CounterService.Iface
>
> In the PHP Client code I have,
>
> --Client.php--
> $server = '192.168.1.64';
> $socket = new TSocket($server, 8888, TRUE); //Using persistent connection
> to
> the Java Service
> $transport = new TBufferedTransport($socket, 1024, 1024);
> $protocol = new TBinaryProtocol($transport);
> $client = new CounterServiceClient($protocol);
> $transport->open();
> $jcount = $client->increment($auctionId);
> $transport->close();
> ---
>
> The problem is the fact that; under our production load (1500 persistent
> socket connections from 7 PHP clients, about a total of 300 requests per
> second), the PHP Client fails to get the correct increment(id) value from
> the Buffer, it somehow gets previous replies from the Server.
>
> I'm logging the Java Server transactions, the Java Server seems to work
> fine; I'm suspecting that the problem occurs because of the PHP client
> libraries.
>
> We are running PHP in lighttpd and fastcgi btw.
>
> Any comments on solving the problem is welcome.
>
> Regards,
> Utku
>
Re: PHP Client fails to read correct return values from the Java
Service
Posted by Todd Lipcon <to...@cloudera.com>.
On Thu, Jan 14, 2010 at 6:36 PM, Mark Slee <ms...@facebook.com> wrote:
> Would need more info to debug this. Do you mind sharing more of your server
> implementation code? The client code looks fine to me. I'm not really clear
> on how you'd get the result of an old call when you're using a new socket
> each time, unless there is a bug with PHP's underlying persistent socket
> implementation leaving old data in the socket buffer (I've never seen this
> before, and it seems like it would create major problems if this could ever
> happen -- I would have assumed that pfsockopen() would always reset the recv
> buffer).
>
>
Isn't it impossible/racey for PHP to guarantee this? That is to say, it can
*think* the buffer's clear, then hand off the socket to the client, and then
the packets arrive.
-Todd
> Can you try running this without using PHP persistent sockets? See if it
> fixes the issue to open/close a new socket every time?
>
> -----Original Message-----
> From: Utku Can Topçu [mailto:utku@topcu.gen.tr]
> Sent: Tuesday, January 05, 2010 12:02 AM
> To: thrift-user@incubator.apache.org
> Subject: PHP Client fails to read correct return values from the Java
> Service
>
> Dear Thrift users and Developers,
>
> I've written a simple service which has the following thrift definition:
>
> service CounterService {
> string increment(1:i64 itemId),
> string getCount(1:i64 itemId),
> }
>
> In our system, we're using PHP as the client and Java as the Server,
> Following the sample client and server stub examples included in the
> tutorial package, I haven't made any big differences in both the Client and
> Server code (and of course never touched the auto-generated code);
>
> The Java Server makes use of the following simple stub:
>
> --Server.java--
> CounterHandler handler = new CounterHandler();
> CounterService.Processor processor = new CounterService.Processor(handler);
> TServerTransport serverTransport;
> serverTransport = new TServerSocket(8888);
> server = new TThreadPoolServer(processor, serverTransport);
> ---
>
> The CounterHandler implements CounterService.Iface
>
> In the PHP Client code I have,
>
> --Client.php--
> $server = '192.168.1.64';
> $socket = new TSocket($server, 8888, TRUE); //Using persistent connection
> to
> the Java Service
> $transport = new TBufferedTransport($socket, 1024, 1024);
> $protocol = new TBinaryProtocol($transport);
> $client = new CounterServiceClient($protocol);
> $transport->open();
> $jcount = $client->increment($auctionId);
> $transport->close();
> ---
>
> The problem is the fact that; under our production load (1500 persistent
> socket connections from 7 PHP clients, about a total of 300 requests per
> second), the PHP Client fails to get the correct increment(id) value from
> the Buffer, it somehow gets previous replies from the Server.
>
> I'm logging the Java Server transactions, the Java Server seems to work
> fine; I'm suspecting that the problem occurs because of the PHP client
> libraries.
>
> We are running PHP in lighttpd and fastcgi btw.
>
> Any comments on solving the problem is welcome.
>
> Regards,
> Utku
>
RE: PHP Client fails to read correct return values from the Java
Service
Posted by Mark Slee <ms...@facebook.com>.
Would need more info to debug this. Do you mind sharing more of your server implementation code? The client code looks fine to me. I'm not really clear on how you'd get the result of an old call when you're using a new socket each time, unless there is a bug with PHP's underlying persistent socket implementation leaving old data in the socket buffer (I've never seen this before, and it seems like it would create major problems if this could ever happen -- I would have assumed that pfsockopen() would always reset the recv buffer).
Can you try running this without using PHP persistent sockets? See if it fixes the issue to open/close a new socket every time?
-----Original Message-----
From: Utku Can Topçu [mailto:utku@topcu.gen.tr]
Sent: Tuesday, January 05, 2010 12:02 AM
To: thrift-user@incubator.apache.org
Subject: PHP Client fails to read correct return values from the Java Service
Dear Thrift users and Developers,
I've written a simple service which has the following thrift definition:
service CounterService {
string increment(1:i64 itemId),
string getCount(1:i64 itemId),
}
In our system, we're using PHP as the client and Java as the Server,
Following the sample client and server stub examples included in the
tutorial package, I haven't made any big differences in both the Client and
Server code (and of course never touched the auto-generated code);
The Java Server makes use of the following simple stub:
--Server.java--
CounterHandler handler = new CounterHandler();
CounterService.Processor processor = new CounterService.Processor(handler);
TServerTransport serverTransport;
serverTransport = new TServerSocket(8888);
server = new TThreadPoolServer(processor, serverTransport);
---
The CounterHandler implements CounterService.Iface
In the PHP Client code I have,
--Client.php--
$server = '192.168.1.64';
$socket = new TSocket($server, 8888, TRUE); //Using persistent connection to
the Java Service
$transport = new TBufferedTransport($socket, 1024, 1024);
$protocol = new TBinaryProtocol($transport);
$client = new CounterServiceClient($protocol);
$transport->open();
$jcount = $client->increment($auctionId);
$transport->close();
---
The problem is the fact that; under our production load (1500 persistent
socket connections from 7 PHP clients, about a total of 300 requests per
second), the PHP Client fails to get the correct increment(id) value from
the Buffer, it somehow gets previous replies from the Server.
I'm logging the Java Server transactions, the Java Server seems to work
fine; I'm suspecting that the problem occurs because of the PHP client
libraries.
We are running PHP in lighttpd and fastcgi btw.
Any comments on solving the problem is welcome.
Regards,
Utku