You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by Eric Evans <ee...@rackspace.com> on 2010/11/19 02:10:42 UTC
Python SocketTransceiver
I put together a SocketTransceiver for Python so that I could compare
performance against the HTTP one, but I'm having some trouble getting it
to work. I'm suspect it's something really dumb, that I'm just too
close to spot it, and that someone else might.
The patch against 1.4 is attached. It causes the Java server to produce
the following exception:
java.io.EOFException
at
org.apache.avro.ipc.ByteBufferInputStream.getBuffer(ByteBufferInputStream.java:84)
at
org.apache.avro.ipc.ByteBufferInputStream.read(ByteBufferInputStream.java:46)
at org.apache.avro.io.BinaryDecoder
$InputStreamByteSource.readRaw(BinaryDecoder.java:815)
at org.apache.avro.io.BinaryDecoder.doReadBytes(BinaryDecoder.java:340)
at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:265)
at
org.apache.avro.io.ValidatingDecoder.readString(ValidatingDecoder.java:99)
at
org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:318)
at
org.apache.avro.generic.GenericDatumReader.readMap(GenericDatumReader.java:229)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:117)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:105)
at org.apache.avro.ipc.Responder.respond(Responder.java:112)
at org.apache.avro.ipc.SocketServer
$Connection.run(SocketServer.java:91)
at java.lang.Thread.run(Thread.java:636)
Anyone?
--
Eric Evans
eevans@rackspace.com
Re: Python SocketTransceiver
Posted by Doug Cutting <cu...@apache.org>.
On 11/19/2010 12:49 AM, Stephen Gargan wrote:
> In the java
> versions of Requestor and Responder the negotiated protocol is stored
> along with the connection. The connection is considered 'established'
> and so long as it persists, neither side expects the handshake to
> appear in any subsequent call between them.
The other substantial difference between socket-based RPC and http-based
RPC is the handling of one-way messages.
With http, some response data is always written. The response data for
a message that returns void still has, in addition to the handshake
response, an error flag and potentially an error object. (This permitted
one-way to be added without incompatibly changing the specification.)
With sockets, once a session has been established, a one-way message has
no response data at all. So a client can return immediately after
writing the request, in a "fire and forget" style.
This is described at:
http://avro.apache.org/docs/current/spec.html#Call+Format
HTTP is considered "stateless" and sockets are "stateful".
Doug
Re: Python SocketTransceiver
Posted by Eric Evans <ee...@rackspace.com>.
On Fri, 2010-11-19 at 00:49 -0800, Stephen Gargan wrote:
> In order for your python SocketTranciever to work with the Java
> version, it will similarly need to skip sending the handshake once the
> connection is established.
Now I feel stupid. I actually knew this, it was one of the reasons I
was expecting to see better performance. I just completely failed to
make that connection here. :)
> Hope this helps.
It absolutely does. Thanks a bunch, I really appreciate it.
--
Eric Evans
eevans@rackspace.com
Re: Python SocketTransceiver
Posted by Stephen Gargan <st...@gmail.com>.
Eric,
There is a subtle difference between how the ipc works in the http and
socket cases and it boils down to the part the connection plays in
protocol negotiation and the fact that the connection is being reused
in the case of the SocketTranciever and not with the HttpTranciever.
Ideally speaking, once the Requestor and the Responder have negotiated
a protocol between them (the handshake has been completed and a
protocol has been agreed), then so long as the connection is persisted
it is redundant to keep sending the handshake data. In the java
versions of Requestor and Responder the negotiated protocol is stored
along with the connection. The connection is considered 'established'
and so long as it persists, neither side expects the handshake to
appear in any subsequent call between them. This is all
straightforward enough and works perfectly for the Socket case where
the connection is long lived and maintained across invocations.
In the HTTP case, by the nature of Http, the connection exists only
for a single request and response. There may be http keepalive of the
underlying socket, but conceptually the http connection is closed
after the server responds. Because there is no persistent connection
to associate the negotiated protocol with, a handshake ends up being
sent with every request, though admittedly only hashes are transferred
in the subsequent invocations.
>From what I can tell of the Python requestor, its sending the
handshake every time and while this will work with the HttpServer for
the reason above, it will fail for the SocketServer. In this case, on
the second and subequent invocations, the SocketServer thinks that the
protocol negotiation is complete and is expecting to read the specific
payload data but is instead encountering the retransmitted handshake
header. The error it displays is what you're seeing is what happens
when the server expects to read one thing (like a handshake or
connection metadata) but encounters something else.
In order for your python SocketTranciever to work with the Java
version, it will similarly need to skip sending the handshake once the
connection is established.
Hope this helps.
Regards,
Stephen.
On Thu, Nov 18, 2010 at 5:10 PM, Eric Evans <ee...@rackspace.com> wrote:
>
> I put together a SocketTransceiver for Python so that I could compare
> performance against the HTTP one, but I'm having some trouble getting it
> to work. I'm suspect it's something really dumb, that I'm just too
> close to spot it, and that someone else might.
>
> The patch against 1.4 is attached. It causes the Java server to produce
> the following exception:
>
> java.io.EOFException
> at
> org.apache.avro.ipc.ByteBufferInputStream.getBuffer(ByteBufferInputStream.java:84)
> at
> org.apache.avro.ipc.ByteBufferInputStream.read(ByteBufferInputStream.java:46)
> at org.apache.avro.io.BinaryDecoder
> $InputStreamByteSource.readRaw(BinaryDecoder.java:815)
> at org.apache.avro.io.BinaryDecoder.doReadBytes(BinaryDecoder.java:340)
> at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:265)
> at
> org.apache.avro.io.ValidatingDecoder.readString(ValidatingDecoder.java:99)
> at
> org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:318)
> at
> org.apache.avro.generic.GenericDatumReader.readMap(GenericDatumReader.java:229)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:117)
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:105)
> at org.apache.avro.ipc.Responder.respond(Responder.java:112)
> at org.apache.avro.ipc.SocketServer
> $Connection.run(SocketServer.java:91)
> at java.lang.Thread.run(Thread.java:636)
>
> Anyone?
>
> --
> Eric Evans
> eevans@rackspace.com
>