You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@crunch.apache.org by Samik Raychaudhuri <sa...@gmail.com> on 2014/09/11 14:41:45 UTC

Getting to PCollection of specific class

Hello,
Newbie alert :-)
I am trying to get a PCollection of the class which is a POJO generated 
from my avro schema. Here is what I have:

AvroMode.setSpecificClassLoader(Event.class.getClassLoader());
         PCollection<Event> records = 
pipeline.read(From.avroFile("/raw/*.avro", Avros.specifics(Event.class)));
         pipeline.materialize(records);

The Event class above is generated from the avro schema. I am getting 
the following exception:

Exception in thread "main" java.lang.NoSuchMethodError: 
org.apache.avro.Schema$Parser.parse(Ljava/lang/String;[Ljava/lang/String;)Lorg/apache/avro/Schema;
     at com.tfs.interaction.event.avro.Event.<clinit>(Event.java:10)
     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
Method)
     at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
     at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
     at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
     at 
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:129)
     at org.apache.crunch.types.avro.Avros.specifics(Avros.java:267)
*at 
com.ilabs.dsi.dataQuality.DataQualityChecker.run(DataQualityChecker.java:50)*
     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
     at 
com.ilabs.dsi.dataQuality.DataQualityChecker.main(DataQualityChecker.java:81)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
     at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke(Method.java:483)
     at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)

The line from where the Exception starts is the second line in the 
snippet above. I essentially tried reusing the snippet that is mentioned 
here: https://issues.apache.org/jira/browse/CRUNCH-442

When I load the data using the following snippet to Record objects, 
things work fine:

         Schema eventSchema = new Schema.Parser().parse(new 
File(schemaPath));
         System.out.println("Loaded IDM schema.");
         PCollection<Record> events = 
pipeline.read(From.avroFile("/raw/*.avro", Avros.generics(eventSchema)));


What am I doing wrong?
Regards.

Re: Getting to PCollection of specific class

Posted by Samik Raychaudhuri <sa...@gmail.com>.

Ah .. turned out to be a rather easy fix - I had to upgrade avro library 
version from 1.7.4 to the most recent one (1.7.7).
Thanks.

On 11/09/2014 6:11 PM, Samik Raychaudhuri wrote:
> Hello,
> Newbie alert :-)
> I am trying to get a PCollection of the class which is a POJO 
> generated from my avro schema. Here is what I have:
>
> AvroMode.setSpecificClassLoader(Event.class.getClassLoader());
>         PCollection<Event> records = 
> pipeline.read(From.avroFile("/raw/*.avro", Avros.specifics(Event.class)));
>         pipeline.materialize(records);
>
> The Event class above is generated from the avro schema. I am getting 
> the following exception:
>
> Exception in thread "main" java.lang.NoSuchMethodError: 
> org.apache.avro.Schema$Parser.parse(Ljava/lang/String;[Ljava/lang/String;)Lorg/apache/avro/Schema;
>     at com.tfs.interaction.event.avro.Event.<clinit>(Event.java:10)
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
>     at 
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:129)
>     at org.apache.crunch.types.avro.Avros.specifics(Avros.java:267)
> *at 
> com.ilabs.dsi.dataQuality.DataQualityChecker.run(DataQualityChecker.java:50)*
>     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>     at 
> com.ilabs.dsi.dataQuality.DataQualityChecker.main(DataQualityChecker.java:81)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:483)
>     at 
> com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
>
> The line from where the Exception starts is the second line in the 
> snippet above. I essentially tried reusing the snippet that is 
> mentioned here: https://issues.apache.org/jira/browse/CRUNCH-442
>
> When I load the data using the following snippet to Record objects, 
> things work fine:
>
>         Schema eventSchema = new Schema.Parser().parse(new 
> File(schemaPath));
>         System.out.println("Loaded IDM schema.");
>         PCollection<Record> events = 
> pipeline.read(From.avroFile("/raw/*.avro", Avros.generics(eventSchema)));
>
>
> What am I doing wrong?
> Regards.