You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Deepak Nettem <de...@gmail.com> on 2012/03/29 21:54:57 UTC
AvroMapper map method not getting invoked
HI,
I am trying to run a simple Avro MapReduce Mapper only job, that reads from
an Avro source. This piece of code simply reads my avro source data and
writes it to the outpath in the exact same schema, irrespective of what is
given in the map method of the mapper. The map method is not getting
invoked at all.
Why is this happening?
-----------------
public class AvroReader {
//public static final Schema schema;
public static final Schema IN_SCHEMA =
ReflectData.get().getSchema(Record.class);
public void AvroReaderRunner(String inputPath, String outputPath) throws
Exception {
JobConf conf = new JobConf();
Job job = new Job(conf,"AvroRecord Job..");
job.setJobName("avroreader");
job.setJarByClass(AvroReader.class);
AvroJob.setInputSchema(conf, IN_SCHEMA);
//Schema stringSchema = Schema.create(Schema.Type.STRING);
//Schema pairSchema = Pair.getPairSchema(stringSchema, stringSchema);
AvroJob.setMapOutputSchema(conf, new Pair<Utf8,Utf8>(new Utf8(""),
0L).getSchema() );
//AvroJob.setOutputSchema(conf, IN_SCHEMA.)
AvroJob.setMapperClass(conf, AvroReaderMapper.class);
FileInputFormat.addInputPath(job, new Path(inputPath));
//FileInputFormat.addInputPath(job, new Path(inputPath));
FileOutputFormat.setOutputPath(job, new Path(outputPath));
//FileOutputFormat.setOutputPath(job, new Path(outputPath));
job.setNumReduceTasks(0); // map-only
job.waitForCompletion(true);
}
public static class AvroReaderMapper extends AvroMapper<Record,
Pair<Utf8, Utf8>>{
@Override
public void map(Record in, AvroCollector<Pair<Utf8,Utf8>> collector,
Reporter reporter) throws IOException {
collector.collect(new Pair<Utf8,Utf8>('a', 'b'));
}
}
}
--
Warm Regards,
Deepak Nettem
Re: AvroMapper map method not getting invoked
Posted by Jeremy Lewi <je...@lewi.us>.
One thing I've run into is not properly overriding the map and reduce
method.
The AvroMapper (
http://svn.apache.org/viewvc/avro/trunk/lang/java/mapred/src/main/java/org/apache/avro/mapred/AvroMapper.java?view=markup
)
defines a default implementation for map. So if you haven't properly
overloaded this method then your map function won't get invoked and you'll
wonder why nothing is happening when really the default implementation is
being invoked and it does nothing.
An easy way to check this by adding @Override to your map function. If your
method doesn't match an inherited function this should cause a compiler
warning.
J
On Thu, Mar 29, 2012 at 1:42 PM, Serge Blazhievsky <ea...@gmail.com>wrote:
> Can you post the code for your main method?
>
> Serge
>
>
> On Thu, Mar 29, 2012 at 12:54 PM, Deepak Nettem <de...@gmail.com>wrote:
>
>> HI,
>>
>> I am trying to run a simple Avro MapReduce Mapper only job, that reads
>> from an Avro source. This piece of code simply reads my avro source data
>> and writes it to the outpath in the exact same schema, irrespective of what
>> is given in the map method of the mapper. The map method is not getting
>> invoked at all.
>>
>> Why is this happening?
>>
>> -----------------
>> public class AvroReader {
>>
>> //public static final Schema schema;
>> public static final Schema IN_SCHEMA =
>> ReflectData.get().getSchema(Record.class);
>>
>> public void AvroReaderRunner(String inputPath, String outputPath)
>> throws Exception {
>>
>> JobConf conf = new JobConf();
>>
>> Job job = new Job(conf,"AvroRecord Job..");
>> job.setJobName("avroreader");
>> job.setJarByClass(AvroReader.class);
>>
>> AvroJob.setInputSchema(conf, IN_SCHEMA);
>>
>> //Schema stringSchema = Schema.create(Schema.Type.STRING);
>> //Schema pairSchema = Pair.getPairSchema(stringSchema, stringSchema);
>>
>> AvroJob.setMapOutputSchema(conf, new Pair<Utf8,Utf8>(new Utf8(""),
>> 0L).getSchema() );
>>
>> //AvroJob.setOutputSchema(conf, IN_SCHEMA.)
>> AvroJob.setMapperClass(conf, AvroReaderMapper.class);
>>
>> FileInputFormat.addInputPath(job, new Path(inputPath));
>>
>> //FileInputFormat.addInputPath(job, new Path(inputPath));
>> FileOutputFormat.setOutputPath(job, new Path(outputPath));
>>
>> //FileOutputFormat.setOutputPath(job, new Path(outputPath));
>>
>> job.setNumReduceTasks(0); // map-only
>>
>> job.waitForCompletion(true);
>> }
>>
>> public static class AvroReaderMapper extends AvroMapper<Record,
>> Pair<Utf8, Utf8>>{
>>
>> @Override
>> public void map(Record in, AvroCollector<Pair<Utf8,Utf8>> collector,
>> Reporter reporter) throws IOException {
>> collector.collect(new Pair<Utf8,Utf8>('a', 'b'));
>>
>> }
>>
>> }
>>
>> }
>>
>>
>> --
>> Warm Regards,
>> Deepak Nettem
>>
>>
>>
>
Re: AvroMapper map method not getting invoked
Posted by Serge Blazhievsky <ea...@gmail.com>.
Can you post the code for your main method?
Serge
On Thu, Mar 29, 2012 at 12:54 PM, Deepak Nettem <de...@gmail.com>wrote:
> HI,
>
> I am trying to run a simple Avro MapReduce Mapper only job, that reads
> from an Avro source. This piece of code simply reads my avro source data
> and writes it to the outpath in the exact same schema, irrespective of what
> is given in the map method of the mapper. The map method is not getting
> invoked at all.
>
> Why is this happening?
>
> -----------------
> public class AvroReader {
>
> //public static final Schema schema;
> public static final Schema IN_SCHEMA =
> ReflectData.get().getSchema(Record.class);
>
> public void AvroReaderRunner(String inputPath, String outputPath) throws
> Exception {
>
> JobConf conf = new JobConf();
>
> Job job = new Job(conf,"AvroRecord Job..");
> job.setJobName("avroreader");
> job.setJarByClass(AvroReader.class);
>
> AvroJob.setInputSchema(conf, IN_SCHEMA);
>
> //Schema stringSchema = Schema.create(Schema.Type.STRING);
> //Schema pairSchema = Pair.getPairSchema(stringSchema, stringSchema);
>
> AvroJob.setMapOutputSchema(conf, new Pair<Utf8,Utf8>(new Utf8(""),
> 0L).getSchema() );
>
> //AvroJob.setOutputSchema(conf, IN_SCHEMA.)
> AvroJob.setMapperClass(conf, AvroReaderMapper.class);
>
> FileInputFormat.addInputPath(job, new Path(inputPath));
>
> //FileInputFormat.addInputPath(job, new Path(inputPath));
> FileOutputFormat.setOutputPath(job, new Path(outputPath));
>
> //FileOutputFormat.setOutputPath(job, new Path(outputPath));
>
> job.setNumReduceTasks(0); // map-only
>
> job.waitForCompletion(true);
> }
>
> public static class AvroReaderMapper extends AvroMapper<Record,
> Pair<Utf8, Utf8>>{
>
> @Override
> public void map(Record in, AvroCollector<Pair<Utf8,Utf8>> collector,
> Reporter reporter) throws IOException {
> collector.collect(new Pair<Utf8,Utf8>('a', 'b'));
>
> }
>
> }
>
> }
>
>
> --
> Warm Regards,
> Deepak Nettem
>
>
>