avro-tools tojson where avro file in HDFS

["Ashish Rastogi (BLOOMBERG/ 731 LEX)" <ar...@bloomberg.net>]

Hi Avro users,

I'm using avro-tools-1.7.7.jar, and would like to print records to stdout using the "tojson" option. I want to do this with my avro files in HDFS (and not on the local file system). I thought AVRO-867 (https://issues.apache.org/jira/browse/AVRO-867) would allow me to do this. However, I get the following exception when I run: 

$ hadoop jar avro-tools-1.7.7.jar tojson hdfs://<avro file location>

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.avro.io.EncoderFactory.jsonEncoder(Lorg/apache/avro/Schema;Ljava/io/OutputStream;Z)Lorg/apache/avro/io/JsonEncoder;
at org.apache.avro.tool.DataFileReadTool.run(DataFileReadTool.java:76)
at org.apache.avro.tool.Main.run(Main.java:84)
at org.apache.avro.tool.Main.main(Main.java:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Anyone knows what this means? Any help is appreciated.

As a confirmation, I'm able to use avro-tools after I -copyToLocal this avro file to view these records.

Thanks,
Ashish

Re: avro-tools tojson where avro file in HDFS

[Harsh J <ha...@cloudera.com>]

The avro-tools jar is usually a standalone one, and if you do have a
standalone variant running it with 'hadoop jar' may cause classpath
pollution as hadoop also includes a (likely different) version of avro into
the runtime classpath.

Run it instead this way:

export HADOOP_USER_CLASSPATH_FIRST=true
export HADOOP_CLASSPATH=avro-tools-1.7.7.jar
hadoop jar avro-tools-1.7.7.jar …

Or if you are certain avro-tools is built against a compatible hadoop
server version, simply run:

java -jar avro-tools-1.7.7.jar …

On Fri, Sep 4, 2015 at 8:26 PM Ashish Rastogi (BLOOMBERG/ 731 LEX) <
arastogi32@bloomberg.net> wrote:

> Hi Avro users,
>
> I'm using avro-tools-1.7.7.jar, and would like to print records to stdout
> using the "tojson" option. I want to do this with my avro files in HDFS
> (and not on the local file system). I thought AVRO-867 (
> https://issues.apache.org/jira/browse/AVRO-867) would allow me to do
> this. However, I get the following exception when I run:
>
> $ hadoop jar avro-tools-1.7.7.jar tojson hdfs://<avro file location>
>
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.avro.io.EncoderFactory.jsonEncoder(Lorg/apache/avro/Schema;Ljava/io/OutputStream;Z)Lorg/apache/avro/io/JsonEncoder;
> at org.apache.avro.tool.DataFileReadTool.run(DataFileReadTool.java:76)
> at org.apache.avro.tool.Main.run(Main.java:84)
> at org.apache.avro.tool.Main.main(Main.java:73)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>
> Anyone knows what this means? Any help is appreciated.
>
> As a confirmation, I'm able to use avro-tools after I -copyToLocal this
> avro file to view these records.
>
> Thanks,
> Ashish
>