java.lang.NoSuchMethodError: org.apache.avro.Schema.access$1400()Ljava/lang/ThreadLocal

[Jesse Jaggars <jh...@gmail.com>]

I'm trying to build a simple mapreduce job that reads avro files and outputs plain text.

I pulled data from a mysql database with sqoop and wrote the files out as snappy compressed avro files. I've operated on the files using AvroStorage in pig, but the current task I'm trying to accomplish would be better suited with a plain MR job I think. I'm using avro 1.7.4 but I think the version used to generate the files is 1.5.3 (whatever ships with HDP 1.2). I've tried depending on avro 1.5.3 in my project but I get the same error (but a different line).

When I try to execute my job the following exception is printed:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.avro.Schema.access$1400()Ljava/lang/ThreadLocal;
        at org.apache.avro.Schema$Parser.parse(Schema.java:924)
        at org.apache.avro.Schema$Parser.parse(Schema.java:917)
        at my.classpath.jobs.CandidatesFor.run(CandidatesFor.java:44)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at my.classpath.jobs.CandidatesFor.main(CandidatesFor.java:72)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:160)

Below is the setup portion of my mapreduce job. The failure occurs when I call parse on the Schema object.

public class CandidatesFor extends Configured implements Tool {

    @Override
    public int run(String[] args) throws Exception {
        JobConf conf = new JobConf(getConf(), getClass());
        conf.setJobName("CandidatesFor");

        InputStream is = getClass().getClassLoader().getResourceAsStream("avro/data.avsc");
        assert null != is;
        String schemaString = IOUtils.toString(is);
        System.out.println(schemaString);
        Schema.Parser parser = new Schema.Parser();
        Schema schema = parser.parse(schemaString); // exception is thrown here
	…

Looking in the code for Schema the line is:

	boolean saved = validateNames.get();

I wrote a unit test to help me understand how things hang together. The test passes:

    @Test
    public void canReadSchema() throws Exception {
        ClassLoader loader = getClass().getClassLoader();
        InputStream schema_is = loader.getResourceAsStream("avro/data.avsc");
        File data = new File(loader.getResource("avro/part-m-00000.avro").toURI());
        assertNotNull(data);
        Schema schema = new Schema.Parser().parse(schema_is);
        DatumReader<GenericRecord> datumReader = new GenericDatumReader<GenericRecord>(schema);
        DataFileReader<GenericRecord> dataFileReader = new DataFileReader<GenericRecord>(data, datumReader);
        GenericRecord case_ = null;
        while (dataFileReader.hasNext()) {
            case_ = dataFileReader.next(case_);
            assertNotNull(case_.get("subject"));
            assertNotNull(case_.get("description"));
        }
    }

I'm sure I'm missing something obvious but I don't know enough to recognize it. Any help would be greatly appreciated. 
Thanks