You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by "R. Tyler Ballance" <ty...@monkeypox.org> on 2010/04/14 22:21:46 UTC
Using avro with hadoop streaming
Howdy folks, I was following the progress of JIRA #493 [1] which has since been
committed but I can not for the life of me get it to cooperate, despite having
recent builds from trunk of the avro jars [2]
I've not seen any chatter on the mailing list or the IRC channel regarding the
feature as of late, so I have a couple questions (probably more for cutting
than anybody):
* Is hadoop streaming support actually /working/ in trunk?
* Are there plans to update the wiki or any other documentation on how to
properly use it?
* If I asked nicely enough, could somebody provide the aforementioned
documentation? I r teh noobs
[1] https://issues.apache.org/jira/browse/AVRO-493
[2] http://hudson.brokenco.de/job/Avro_Trunk/
Cheers,
-R. Tyler Ballance
--------------------------------------
Jabber: rtyler@jabber.org
GitHub: http://github.com/rtyler
Twitter: http://twitter.com/agentdero
Blog: http://unethicalblogger.com
Re: Using avro with hadoop streaming
Posted by "R. Tyler Ballance" <ty...@monkeypox.org>.
On Wed, 21 Apr 2010, Doug Cutting wrote:
> R. Tyler Ballance wrote:
> >Is hadoop streaming support actually /working/ in trunk?
>
> Hadoop Streaming access to Avro data? No. Hadoop Streaming is
> primarily intended for textual, CSV-style data.
>
> To better integrate languages Avro data into Perl, Python and Ruby
> mapreduce programs, we hope to builds something like Hadoop Pipes.
>
> https://issues.apache.org/jira/browse/AVRO-512
>
> I hope to work on this in the coming weeks.
Ah, this rings a bit clearer to me, mind you I'm a hadidiot, I'm more
into generating the avro datas (and the RPC!).
I'll follow the ticket, looking forward to seeing that going in.
>
> AVRO-493 only provides Avro data to Java mapreduce programs. The
> best documentation for it currently are its unit test source code.
>
> http://tinyurl.com/yz8bd22
> http://tinyurl.com/2a3xbu8
Handy links, I don't think we're going to invest any time in writing anything
other than Python code for the time being. Until you have the chance to crank
through #512, our intermediary solution has been to pre-process avro logs,
pulling out the schema into a separate file and dumping it to a textual JSON
file suitable for streaming into hadoop.
Cheers,
-R. Tyler Ballance
--------------------------------------
Jabber: rtyler@jabber.org
GitHub: http://github.com/rtyler
Identica: http://identi.ca/dero
Twitter: http://twitter.com/agentdero
Blog: http://unethicalblogger.com
Re: Using avro with hadoop streaming
Posted by Doug Cutting <cu...@apache.org>.
R. Tyler Ballance wrote:
> Is hadoop streaming support actually /working/ in trunk?
Hadoop Streaming access to Avro data? No. Hadoop Streaming is
primarily intended for textual, CSV-style data.
To better integrate languages Avro data into Perl, Python and Ruby
mapreduce programs, we hope to builds something like Hadoop Pipes.
https://issues.apache.org/jira/browse/AVRO-512
I hope to work on this in the coming weeks.
AVRO-493 only provides Avro data to Java mapreduce programs. The best
documentation for it currently are its unit test source code.
http://tinyurl.com/yz8bd22
http://tinyurl.com/2a3xbu8
Doug