You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Something Something <ma...@gmail.com> on 2011/10/07 08:42:27 UTC
Reading AVRO files in Java MapReduce
Would like to read & process AVRO files in a Java MapReduce program. I
googled but didn't see any good examples that I could use. If anyone can
provide samples, that would be greatly appreciated. Thanks.
Re: Reading AVRO files in Java MapReduce
Posted by Something Something <ma...@gmail.com>.
Okay.... I think I am down to my last problem. I have a 'AvroMapper' & a
'NonAvroReducer'. The Reducer is defined as follows:
private static class NonAvroReducer
extends MapReduceBase
implements Reducer<AvroKey<Utf8>, AvroValue<MyClass>, Text,
Text> {
I keep getting this in the Reduce step:
java.lang.ClassCastException: com.xyz.MyAvroProcessor$MyClass cannot be
cast to org.apache.avro.generic.IndexedRecord
at org.apache.avro.generic.GenericData.setField(GenericData.java:463)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:166)
But 'MyClass' is just a Java Bean that I want to use to pass data between
Mapper & Reducer. Why do I have to implement IndexedRecord?
Please help. Thanks.
On Wed, Nov 16, 2011 at 11:20 PM, Something Something <
mailinglists19@gmail.com> wrote:
> Sorry. Don't worry about this for now. Made progress. The input AVRO
> file was bad. Thanks.
>
>
> On Wed, Nov 16, 2011 at 11:13 PM, Sudharsan Sampath <su...@gmail.com>wrote:
>
>> What's the output logs from your job..
>>
>>
>> On Thu, Nov 17, 2011 at 5:30 AM, Something Something <
>> mailinglists19@gmail.com> wrote:
>>
>>> Why could this not be working? Any ideas? I even put a 'throw new
>>> RuntimeException' to see if it's coming to the Mapper, but it isn't.
>>> Thanks for the help.
>>>
>>> Code snippet:
>>>
>>>
>>>
>>> public static class MapImpl extends AvroMapper<Utf8, Pair<Utf8,
>>> Long>> {
>>>
>>> @Override
>>>
>>> public void map(Utf8 text, AvroCollector<Pair<Utf8, Long>>
>>> collector,
>>>
>>> Reporter reporter) throws IOException {
>>>
>>> throw new RuntimeException("my text: " + text.toString());
>>>
>>> // System.out.println("my text" + text);
>>>
>>> //
>>>
>>> // collector.collect(new Pair<Utf8, Long>(text, 1L));
>>>
>>> }
>>>
>>> }
>>>
>>>
>>>
>>> private static class NonAvroReducer
>>>
>>> extends MapReduceBase
>>>
>>> implements Reducer<AvroKey<Utf8>, AvroValue<Long>, Text,
>>> Text> {
>>>
>>>
>>> public void reduce(AvroKey<Utf8> key, Iterator<AvroValue<Long>>
>>> values,
>>>
>>> OutputCollector<Text, Text> out,
>>>
>>> Reporter reporter) throws IOException {
>>>
>>> out.collect(new Text(key.toString()),
>>>
>>> new Text("Testing"));
>>>
>>> while (values.hasNext()) {
>>>
>>> AvroValue<Long> value = values.next();
>>>
>>> out.collect(new Text(key.toString()),
>>>
>>> new Text(value.datum().toString()));
>>>
>>> }
>>>
>>> }
>>>
>>> }
>>>
>>>
>>> public static void main(String[] args) throws Exception {
>>>
>>>
>>>
>>> String dir = "/user/mydir";
>>>
>>>
>>>
>>> JobConf job = new JobConf(new Configuration(),
>>> TestAvroProcessor.class);
>>>
>>> job.setJobName(TestAvroProcessor.class.getName());
>>>
>>>
>>>
>>> Path outputPath = new Path(dir + "/out");
>>>
>>>
>>> outputPath.getFileSystem(job).delete(outputPath);
>>>
>>>
>>>
>>> AvroJob.setInputSchema(job, Schema.parse(new File(
>>> "/Users/mydir/profiles.json")));
>>>
>>> AvroJob.setOutputSchema(job, SCHEMA);
>>>
>>>
>>> AvroJob.setMapperClass(job, MapImpl.class);
>>>
>>>
>>> FileInputFormat.setInputPaths(job, new Path(dir + "/data"));
>>>
>>> FileOutputFormat.setOutputPath(job, outputPath);
>>>
>>> FileOutputFormat.setCompressOutput(job, false);
>>>
>>>
>>>
>>> job.setReducerClass(NonAvroReducer.class);
>>>
>>> job.setOutputFormat(TextOutputFormat.class);
>>>
>>> job.setOutputKeyClass(Text.class);
>>>
>>> job.setOutputValueClass(Text.class);
>>>
>>>
>>>
>>> JobClient.runJob(job);
>>>
>>>
>>>
>>>
>>>
>>> }
>>>
>>>
>>>
>>>
>>> On Tue, Nov 15, 2011 at 5:12 PM, Doug Cutting <cu...@apache.org>wrote:
>>>
>>>> On 11/15/2011 03:16 PM, Something Something wrote:
>>>> > Quick question. I want the output from AvroJob.setReducerClass to be
>>>> in
>>>> > regular Text files - not in AVRO format. Can I do that? Any
>>>> examples?
>>>> > Sorry, kinda short on time to do research. Thanks.
>>>>
>>>> On the previously cited documentation page:
>>>>
>>>>
>>>> http://avro.apache.org/docs/current/api/java/org/apache/avro/mapred/package-summary.html
>>>>
>>>> Look for the text, "For jobs whose input is an Avro data file and which
>>>> use an AvroMapper, but whose reducer is a non-Avro Reducer and whose
>>>> output is a non-Avro format".
>>>>
>>>> A sample of a job that does this is at:
>>>>
>>>> http://s.apache.org/MsG
>>>>
>>>> Just use TextOutputFormat instead of SequenceFileOutputFormat.
>>>>
>>>> Doug
>>>>
>>>
>>>
>>
>
Re: Reading AVRO files in Java MapReduce
Posted by Something Something <ma...@gmail.com>.
Sorry. Don't worry about this for now. Made progress. The input AVRO
file was bad. Thanks.
On Wed, Nov 16, 2011 at 11:13 PM, Sudharsan Sampath <su...@gmail.com>wrote:
> What's the output logs from your job..
>
>
> On Thu, Nov 17, 2011 at 5:30 AM, Something Something <
> mailinglists19@gmail.com> wrote:
>
>> Why could this not be working? Any ideas? I even put a 'throw new
>> RuntimeException' to see if it's coming to the Mapper, but it isn't.
>> Thanks for the help.
>>
>> Code snippet:
>>
>>
>>
>> public static class MapImpl extends AvroMapper<Utf8, Pair<Utf8,
>> Long>> {
>>
>> @Override
>>
>> public void map(Utf8 text, AvroCollector<Pair<Utf8, Long>>
>> collector,
>>
>> Reporter reporter) throws IOException {
>>
>> throw new RuntimeException("my text: " + text.toString());
>>
>> // System.out.println("my text" + text);
>>
>> //
>>
>> // collector.collect(new Pair<Utf8, Long>(text, 1L));
>>
>> }
>>
>> }
>>
>>
>>
>> private static class NonAvroReducer
>>
>> extends MapReduceBase
>>
>> implements Reducer<AvroKey<Utf8>, AvroValue<Long>, Text,
>> Text> {
>>
>>
>> public void reduce(AvroKey<Utf8> key, Iterator<AvroValue<Long>>
>> values,
>>
>> OutputCollector<Text, Text> out,
>>
>> Reporter reporter) throws IOException {
>>
>> out.collect(new Text(key.toString()),
>>
>> new Text("Testing"));
>>
>> while (values.hasNext()) {
>>
>> AvroValue<Long> value = values.next();
>>
>> out.collect(new Text(key.toString()),
>>
>> new Text(value.datum().toString()));
>>
>> }
>>
>> }
>>
>> }
>>
>>
>> public static void main(String[] args) throws Exception {
>>
>>
>>
>> String dir = "/user/mydir";
>>
>>
>>
>> JobConf job = new JobConf(new Configuration(), TestAvroProcessor.
>> class);
>>
>> job.setJobName(TestAvroProcessor.class.getName());
>>
>>
>>
>> Path outputPath = new Path(dir + "/out");
>>
>>
>> outputPath.getFileSystem(job).delete(outputPath);
>>
>>
>>
>> AvroJob.setInputSchema(job, Schema.parse(new File(
>> "/Users/mydir/profiles.json")));
>>
>> AvroJob.setOutputSchema(job, SCHEMA);
>>
>>
>> AvroJob.setMapperClass(job, MapImpl.class);
>>
>>
>> FileInputFormat.setInputPaths(job, new Path(dir + "/data"));
>>
>> FileOutputFormat.setOutputPath(job, outputPath);
>>
>> FileOutputFormat.setCompressOutput(job, false);
>>
>>
>>
>> job.setReducerClass(NonAvroReducer.class);
>>
>> job.setOutputFormat(TextOutputFormat.class);
>>
>> job.setOutputKeyClass(Text.class);
>>
>> job.setOutputValueClass(Text.class);
>>
>>
>>
>> JobClient.runJob(job);
>>
>>
>>
>>
>>
>> }
>>
>>
>>
>>
>> On Tue, Nov 15, 2011 at 5:12 PM, Doug Cutting <cu...@apache.org> wrote:
>>
>>> On 11/15/2011 03:16 PM, Something Something wrote:
>>> > Quick question. I want the output from AvroJob.setReducerClass to be
>>> in
>>> > regular Text files - not in AVRO format. Can I do that? Any examples?
>>> > Sorry, kinda short on time to do research. Thanks.
>>>
>>> On the previously cited documentation page:
>>>
>>>
>>> http://avro.apache.org/docs/current/api/java/org/apache/avro/mapred/package-summary.html
>>>
>>> Look for the text, "For jobs whose input is an Avro data file and which
>>> use an AvroMapper, but whose reducer is a non-Avro Reducer and whose
>>> output is a non-Avro format".
>>>
>>> A sample of a job that does this is at:
>>>
>>> http://s.apache.org/MsG
>>>
>>> Just use TextOutputFormat instead of SequenceFileOutputFormat.
>>>
>>> Doug
>>>
>>
>>
>
Re: Reading AVRO files in Java MapReduce
Posted by Sudharsan Sampath <su...@gmail.com>.
What's the output logs from your job..
On Thu, Nov 17, 2011 at 5:30 AM, Something Something <
mailinglists19@gmail.com> wrote:
> Why could this not be working? Any ideas? I even put a 'throw new
> RuntimeException' to see if it's coming to the Mapper, but it isn't.
> Thanks for the help.
>
> Code snippet:
>
>
>
> public static class MapImpl extends AvroMapper<Utf8, Pair<Utf8,
> Long>> {
>
> @Override
>
> public void map(Utf8 text, AvroCollector<Pair<Utf8, Long>>
> collector,
>
> Reporter reporter) throws IOException {
>
> throw new RuntimeException("my text: " + text.toString());
>
> // System.out.println("my text" + text);
>
> //
>
> // collector.collect(new Pair<Utf8, Long>(text, 1L));
>
> }
>
> }
>
>
>
> private static class NonAvroReducer
>
> extends MapReduceBase
>
> implements Reducer<AvroKey<Utf8>, AvroValue<Long>, Text,
> Text> {
>
>
> public void reduce(AvroKey<Utf8> key, Iterator<AvroValue<Long>>
> values,
>
> OutputCollector<Text, Text> out,
>
> Reporter reporter) throws IOException {
>
> out.collect(new Text(key.toString()),
>
> new Text("Testing"));
>
> while (values.hasNext()) {
>
> AvroValue<Long> value = values.next();
>
> out.collect(new Text(key.toString()),
>
> new Text(value.datum().toString()));
>
> }
>
> }
>
> }
>
>
> public static void main(String[] args) throws Exception {
>
>
>
> String dir = "/user/mydir";
>
>
>
> JobConf job = new JobConf(new Configuration(), TestAvroProcessor.
> class);
>
> job.setJobName(TestAvroProcessor.class.getName());
>
>
>
> Path outputPath = new Path(dir + "/out");
>
>
> outputPath.getFileSystem(job).delete(outputPath);
>
>
>
> AvroJob.setInputSchema(job, Schema.parse(new File(
> "/Users/mydir/profiles.json")));
>
> AvroJob.setOutputSchema(job, SCHEMA);
>
>
> AvroJob.setMapperClass(job, MapImpl.class);
>
>
> FileInputFormat.setInputPaths(job, new Path(dir + "/data"));
>
> FileOutputFormat.setOutputPath(job, outputPath);
>
> FileOutputFormat.setCompressOutput(job, false);
>
>
>
> job.setReducerClass(NonAvroReducer.class);
>
> job.setOutputFormat(TextOutputFormat.class);
>
> job.setOutputKeyClass(Text.class);
>
> job.setOutputValueClass(Text.class);
>
>
>
> JobClient.runJob(job);
>
>
>
>
>
> }
>
>
>
>
> On Tue, Nov 15, 2011 at 5:12 PM, Doug Cutting <cu...@apache.org> wrote:
>
>> On 11/15/2011 03:16 PM, Something Something wrote:
>> > Quick question. I want the output from AvroJob.setReducerClass to be in
>> > regular Text files - not in AVRO format. Can I do that? Any examples?
>> > Sorry, kinda short on time to do research. Thanks.
>>
>> On the previously cited documentation page:
>>
>>
>> http://avro.apache.org/docs/current/api/java/org/apache/avro/mapred/package-summary.html
>>
>> Look for the text, "For jobs whose input is an Avro data file and which
>> use an AvroMapper, but whose reducer is a non-Avro Reducer and whose
>> output is a non-Avro format".
>>
>> A sample of a job that does this is at:
>>
>> http://s.apache.org/MsG
>>
>> Just use TextOutputFormat instead of SequenceFileOutputFormat.
>>
>> Doug
>>
>
>
Re: Reading AVRO files in Java MapReduce
Posted by Something Something <ma...@gmail.com>.
Why could this not be working? Any ideas? I even put a 'throw new
RuntimeException' to see if it's coming to the Mapper, but it isn't.
Thanks for the help.
Code snippet:
public static class MapImpl extends AvroMapper<Utf8, Pair<Utf8, Long>> {
@Override
public void map(Utf8 text, AvroCollector<Pair<Utf8, Long>>
collector,
Reporter reporter) throws IOException {
throw new RuntimeException("my text: " + text.toString());
// System.out.println("my text" + text);
//
// collector.collect(new Pair<Utf8, Long>(text, 1L));
}
}
private static class NonAvroReducer
extends MapReduceBase
implements Reducer<AvroKey<Utf8>, AvroValue<Long>, Text, Text> {
public void reduce(AvroKey<Utf8> key, Iterator<AvroValue<Long>>
values,
OutputCollector<Text, Text> out,
Reporter reporter) throws IOException {
out.collect(new Text(key.toString()),
new Text("Testing"));
while (values.hasNext()) {
AvroValue<Long> value = values.next();
out.collect(new Text(key.toString()),
new Text(value.datum().toString()));
}
}
}
public static void main(String[] args) throws Exception {
String dir = "/user/mydir";
JobConf job = new JobConf(new Configuration(), TestAvroProcessor.
class);
job.setJobName(TestAvroProcessor.class.getName());
Path outputPath = new Path(dir + "/out");
outputPath.getFileSystem(job).delete(outputPath);
AvroJob.setInputSchema(job, Schema.parse(new File(
"/Users/mydir/profiles.json")));
AvroJob.setOutputSchema(job, SCHEMA);
AvroJob.setMapperClass(job, MapImpl.class);
FileInputFormat.setInputPaths(job, new Path(dir + "/data"));
FileOutputFormat.setOutputPath(job, outputPath);
FileOutputFormat.setCompressOutput(job, false);
job.setReducerClass(NonAvroReducer.class);
job.setOutputFormat(TextOutputFormat.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
JobClient.runJob(job);
}
On Tue, Nov 15, 2011 at 5:12 PM, Doug Cutting <cu...@apache.org> wrote:
> On 11/15/2011 03:16 PM, Something Something wrote:
> > Quick question. I want the output from AvroJob.setReducerClass to be in
> > regular Text files - not in AVRO format. Can I do that? Any examples?
> > Sorry, kinda short on time to do research. Thanks.
>
> On the previously cited documentation page:
>
>
> http://avro.apache.org/docs/current/api/java/org/apache/avro/mapred/package-summary.html
>
> Look for the text, "For jobs whose input is an Avro data file and which
> use an AvroMapper, but whose reducer is a non-Avro Reducer and whose
> output is a non-Avro format".
>
> A sample of a job that does this is at:
>
> http://s.apache.org/MsG
>
> Just use TextOutputFormat instead of SequenceFileOutputFormat.
>
> Doug
>
Re: Reading AVRO files in Java MapReduce
Posted by Doug Cutting <cu...@apache.org>.
On 11/15/2011 03:16 PM, Something Something wrote:
> Quick question. I want the output from AvroJob.setReducerClass to be in
> regular Text files - not in AVRO format. Can I do that? Any examples?
> Sorry, kinda short on time to do research. Thanks.
On the previously cited documentation page:
http://avro.apache.org/docs/current/api/java/org/apache/avro/mapred/package-summary.html
Look for the text, "For jobs whose input is an Avro data file and which
use an AvroMapper, but whose reducer is a non-Avro Reducer and whose
output is a non-Avro format".
A sample of a job that does this is at:
http://s.apache.org/MsG
Just use TextOutputFormat instead of SequenceFileOutputFormat.
Doug
Re: Reading AVRO files in Java MapReduce
Posted by Something Something <ma...@gmail.com>.
Quick question. I want the output from AvroJob.setReducerClass to be in
regular Text files - not in AVRO format. Can I do that? Any examples?
Sorry, kinda short on time to do research. Thanks.
On Sun, Oct 9, 2011 at 8:49 AM, Something Something <
mailinglists19@gmail.com> wrote:
> Perfect! Thanks a lot, Doug!
>
>
> On Fri, Oct 7, 2011 at 2:06 PM, Doug Cutting <cu...@apache.org> wrote:
>
>> API documentation is at:
>>
>>
>> http://avro.apache.org/docs/current/api/java/org/apache/avro/mapred/package-summary.html
>>
>> Examples may be found in the tests, e.g., from the current release:
>>
>>
>> http://svn.apache.org/viewvc/avro/tags/release-1.5.4/lang/java/mapred/src/test/java/org/apache/avro/mapred/
>>
>> Is that the sort of thing you're looking for?
>>
>> Doug
>>
>> On 10/06/2011 11:42 PM, Something Something wrote:
>> > Would like to read & process AVRO files in a Java MapReduce program. I
>> > googled but didn't see any good examples that I could use. If anyone
>> > can provide samples, that would be greatly appreciated. Thanks.
>>
>
>
Re: Reading AVRO files in Java MapReduce
Posted by Something Something <ma...@gmail.com>.
Perfect! Thanks a lot, Doug!
On Fri, Oct 7, 2011 at 2:06 PM, Doug Cutting <cu...@apache.org> wrote:
> API documentation is at:
>
>
> http://avro.apache.org/docs/current/api/java/org/apache/avro/mapred/package-summary.html
>
> Examples may be found in the tests, e.g., from the current release:
>
>
> http://svn.apache.org/viewvc/avro/tags/release-1.5.4/lang/java/mapred/src/test/java/org/apache/avro/mapred/
>
> Is that the sort of thing you're looking for?
>
> Doug
>
> On 10/06/2011 11:42 PM, Something Something wrote:
> > Would like to read & process AVRO files in a Java MapReduce program. I
> > googled but didn't see any good examples that I could use. If anyone
> > can provide samples, that would be greatly appreciated. Thanks.
>
Re: Reading AVRO files in Java MapReduce
Posted by Doug Cutting <cu...@apache.org>.
API documentation is at:
http://avro.apache.org/docs/current/api/java/org/apache/avro/mapred/package-summary.html
Examples may be found in the tests, e.g., from the current release:
http://svn.apache.org/viewvc/avro/tags/release-1.5.4/lang/java/mapred/src/test/java/org/apache/avro/mapred/
Is that the sort of thing you're looking for?
Doug
On 10/06/2011 11:42 PM, Something Something wrote:
> Would like to read & process AVRO files in a Java MapReduce program. I
> googled but didn't see any good examples that I could use. If anyone
> can provide samples, that would be greatly appreciated. Thanks.