You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Stephen Abrams <Ab...@gmail.com> on 2013/06/20 19:01:47 UTC
Python/Java/Ruby RPC interop issues
Hello!
New to the Avro world. I have been evaluating Flume and testing various
capabilities. I have run into an issue with the Avro RPC piece. So far, I
have found that the Flume Avro RPC java client works with a Flume Avro
Source as expected (I believe these implementations use Netty). However,
when I tried using a Ruby RPC client to communicate with the Flume Avro
Source, there was a failure and the source reported:
org.apache.avro.AvroRuntimeException: Excessively large list allocation
request detected: 539959368 items! Connection closed.
I then tried this with a Python RPC client and had a similar issue. I
suspected client/server Avro interop issues, so then verified that a Python
Avro RPC client and Python Avro RPC server communicate correctly (using
https://github.com/phunt/avro-rpc-quickstart). Also verified that Ruby Avro
RPC client and Ruby Avro RPC server communicate correctly. However, using
the Python Avro RPC client with Ruby Avro RPC server fails (client hangs),
and Ruby client with Python server also fails (again, client hangs).
I know I'm not providing exhaustive info here, but I'm surprised to find
out-of-the-box interop problems. Do folks regularly use these
client/servers in various mixes (Java/Ruby/Python)? Are there particular
gotchas or configurations that are not obvious?
Thank you,
Steve
Re: Python/Java/Ruby RPC interop issues
Posted by Stephen Abrams <Ab...@gmail.com>.
Hi Stefan,
Thanks for the response (and apologies for the delay). The Thrift
workaround (as you say, introduced in 1.4) is an interesting workaround. We
are primarily a Java shop, but some of our tests are written in Python,
which I was using for some quick exploration. If I get ambitious enough I
may try Jython<->Python communication (since I don't want to use Flume for
everything), and post back the results.
Steve
On Mon, Jun 24, 2013 at 8:30 PM, Stefan Krawczyk <st...@nextdoor.com>wrote:
> Hi Steve,
>
> I'm new to the Avro community as well, but it just seems that nobody
> really cares about Avro support outside of Java/C++...
>
> The issue you're seeing is because the python & ruby implementations are
> using the HTTPTransceiver, whereas the Java server is using the
> NettyTransceiver. They dont' talk the same language. I'm as surprised as
> you are that such a change would be introduced without the other languages
> having support (or providing documentation mentioning that they don't work).
>
> I asked this list earlier what it would take to get python to talk netty
> but nobody responded...
>
> We're also using Flume, and the only reasonable option is to use Thrift
> RPC & the Thrift Source (look at 1.4 branch of code) as the ingestion point
> from our app. From there the rest of the flume pipeline can use Avro RPC.
> The other option is to use the legacy avro source, but that source is
> effectively deprecated...
>
> Hope that helps.
>
> Cheers,
>
> Stefan
>
>
> On Thu, Jun 20, 2013 at 10:01 AM, Stephen Abrams <Abrams.Stephen@gmail.com
> > wrote:
>
>> Hello!
>>
>> New to the Avro world. I have been evaluating Flume and testing various
>> capabilities. I have run into an issue with the Avro RPC piece. So far, I
>> have found that the Flume Avro RPC java client works with a Flume Avro
>> Source as expected (I believe these implementations use Netty). However,
>> when I tried using a Ruby RPC client to communicate with the Flume Avro
>> Source, there was a failure and the source reported:
>>
>> org.apache.avro.AvroRuntimeException: Excessively large list allocation
>> request detected: 539959368 items! Connection closed.
>>
>> I then tried this with a Python RPC client and had a similar issue. I
>> suspected client/server Avro interop issues, so then verified that a Python
>> Avro RPC client and Python Avro RPC server communicate correctly (using
>> https://github.com/phunt/avro-rpc-quickstart). Also verified that Ruby
>> Avro RPC client and Ruby Avro RPC server communicate correctly. However,
>> using the Python Avro RPC client with Ruby Avro RPC server fails (client
>> hangs), and Ruby client with Python server also fails (again, client hangs).
>>
>> I know I'm not providing exhaustive info here, but I'm surprised to find
>> out-of-the-box interop problems. Do folks regularly use these
>> client/servers in various mixes (Java/Ruby/Python)? Are there particular
>> gotchas or configurations that are not obvious?
>>
>> Thank you,
>> Steve
>>
>
>
Re: Python/Java/Ruby RPC interop issues
Posted by Stefan Krawczyk <st...@nextdoor.com>.
Hi Steve,
I'm new to the Avro community as well, but it just seems that nobody really
cares about Avro support outside of Java/C++...
The issue you're seeing is because the python & ruby implementations are
using the HTTPTransceiver, whereas the Java server is using the
NettyTransceiver. They dont' talk the same language. I'm as surprised as
you are that such a change would be introduced without the other languages
having support (or providing documentation mentioning that they don't work).
I asked this list earlier what it would take to get python to talk netty
but nobody responded...
We're also using Flume, and the only reasonable option is to use Thrift RPC
& the Thrift Source (look at 1.4 branch of code) as the ingestion point
from our app. From there the rest of the flume pipeline can use Avro RPC.
The other option is to use the legacy avro source, but that source is
effectively deprecated...
Hope that helps.
Cheers,
Stefan
On Thu, Jun 20, 2013 at 10:01 AM, Stephen Abrams
<Ab...@gmail.com>wrote:
> Hello!
>
> New to the Avro world. I have been evaluating Flume and testing various
> capabilities. I have run into an issue with the Avro RPC piece. So far, I
> have found that the Flume Avro RPC java client works with a Flume Avro
> Source as expected (I believe these implementations use Netty). However,
> when I tried using a Ruby RPC client to communicate with the Flume Avro
> Source, there was a failure and the source reported:
>
> org.apache.avro.AvroRuntimeException: Excessively large list allocation
> request detected: 539959368 items! Connection closed.
>
> I then tried this with a Python RPC client and had a similar issue. I
> suspected client/server Avro interop issues, so then verified that a Python
> Avro RPC client and Python Avro RPC server communicate correctly (using
> https://github.com/phunt/avro-rpc-quickstart). Also verified that Ruby
> Avro RPC client and Ruby Avro RPC server communicate correctly. However,
> using the Python Avro RPC client with Ruby Avro RPC server fails (client
> hangs), and Ruby client with Python server also fails (again, client hangs).
>
> I know I'm not providing exhaustive info here, but I'm surprised to find
> out-of-the-box interop problems. Do folks regularly use these
> client/servers in various mixes (Java/Ruby/Python)? Are there particular
> gotchas or configurations that are not obvious?
>
> Thank you,
> Steve
>