You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Tanya Bansal <ta...@gmail.com> on 2012/12/11 04:53:07 UTC
Avro + MRUnit
Are there any examples out there using Apache and MRUnit?
I have configured the AvroSerialization class using the io.serializations
string
I am getting the following exception when I call mapDriver.run():
java.lang.ClassCastException:
amazon.fclm.labormetrics.mapreduce.sample.AvroWordCount$Count cannot be
cast to org.apache.avro.generic.IndexedRecord
at org.apache.avro.generic.GenericData.getField(GenericData.java:518)
at org.apache.avro.generic.GenericData.getField(GenericData.java:533)
at
org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:103)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:57)
at
org.apache.avro.hadoop.io.AvroSerializer.serialize(AvroSerializer.java:103)
at
org.apache.avro.hadoop.io.AvroSerializer.serialize(AvroSerializer.java:45)
at org.apache.hadoop.mrunit.Serialization.copy(Serialization.java:62)
at org.apache.hadoop.mrunit.Serialization.copy(Serialization.java:81)
at
org.apache.hadoop.mrunit.mapreduce.mock.MockContextWrapper$4.answer(MockContextWrapper.java:78)
at
org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
at org.mockito.internal.MockHandler.handle(MockHandler.java:99)
at
org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36)
at
org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48)
at
org.apache.hadoop.mapreduce.Mapper$Context$$EnhancerByMockitoWithCGLIB$$4ae1b65a.write(<generated>)
at
amazon.fclm.labormetrics.mapreduce.sample.AvroWordCount$Map.map(AvroWordCount.java:76)
at
amazon.fclm.labormetrics.mapreduce.sample.AvroWordCount$Map.map(AvroWordCount.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mrunit.mapreduce.MapDriver.run(MapDriver.java:221)
at
amazon.fclm.labormetrics.mapreduce.sample.AvroWordCountTest.TestMrUnitMapper(AvroWordCountTest.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Any help will be appreciated. Thanks!
Re: Avro + MRUnit
Posted by Dave Beech <da...@paraliatech.com>.
Hi Tanya
Sorry I forgot to get back to you about this. When I ran your code I didn't
get the same error as you, but the test didn't produce expected results
either. I think it is because your custom Line and Count classes do not
implement any .equals or compareTo methods to allow them to be compared.
Maybe you could try using built in Avro classes or Java primitives to do
what you need? i.e. Utf8, or String and Integer.
Cheers,
Dave
On 8 January 2013 22:55, Tanya Bansal <ta...@gmail.com> wrote:
> Has anyone else faced the same problem?
>
>
> On Tue, Dec 11, 2012 at 10:38 AM, Tanya Bansal <ta...@gmail.com>wrote:
>
>> Thanks for the quick response.
>>
>> Here is the source code : http://pastebin.com/6t12ZZ5T
>>
>> I have created the Count class on my own.
>>
>> Thanks
>> -Tanya
>>
>>
>>
>>
>>
>>
>> On Tue, Dec 11, 2012 at 1:58 AM, Dave Beech <da...@paraliatech.com> wrote:
>>
>>> Hi Tanya
>>>
>>> How was your Count class created? Did you write it yourself or did you
>>> use Avro tools to generate code from a schema? Something is mixed up
>>> because Avro is trying to write a "generic" record whereas I'd expect it to
>>> be using "specific" records in your case.
>>>
>>> It's hard to work out the problem from just the stacktrace alone, but if
>>> you could share your code (using github gists or pastebin.com), I'd be
>>> happy to take a look.
>>>
>>> Cheers
>>> Dave
>>>
>>>
>>> On 11 December 2012 03:53, Tanya Bansal <ta...@gmail.com> wrote:
>>>
>>>> Are there any examples out there using Apache and MRUnit?
>>>>
>>>> I have configured the AvroSerialization class using the
>>>> io.serializations string
>>>> I am getting the following exception when I call mapDriver.run():
>>>>
>>>> java.lang.ClassCastException:
>>>> amazon.fclm.labormetrics.mapreduce.sample.AvroWordCount$Count cannot be
>>>> cast to org.apache.avro.generic.IndexedRecord
>>>> at org.apache.avro.generic.GenericData.getField(GenericData.java:518)
>>>> at org.apache.avro.generic.GenericData.getField(GenericData.java:533)
>>>> at
>>>> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:103)
>>>> at
>>>> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
>>>> at
>>>> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:57)
>>>> at
>>>> org.apache.avro.hadoop.io.AvroSerializer.serialize(AvroSerializer.java:103)
>>>> at
>>>> org.apache.avro.hadoop.io.AvroSerializer.serialize(AvroSerializer.java:45)
>>>> at org.apache.hadoop.mrunit.Serialization.copy(Serialization.java:62)
>>>> at org.apache.hadoop.mrunit.Serialization.copy(Serialization.java:81)
>>>> at
>>>> org.apache.hadoop.mrunit.mapreduce.mock.MockContextWrapper$4.answer(MockContextWrapper.java:78)
>>>> at
>>>> org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
>>>> at org.mockito.internal.MockHandler.handle(MockHandler.java:99)
>>>> at
>>>> org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36)
>>>> at
>>>> org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48)
>>>> at
>>>> org.apache.hadoop.mapreduce.Mapper$Context$$EnhancerByMockitoWithCGLIB$$4ae1b65a.write(<generated>)
>>>> at
>>>> amazon.fclm.labormetrics.mapreduce.sample.AvroWordCount$Map.map(AvroWordCount.java:76)
>>>> at
>>>> amazon.fclm.labormetrics.mapreduce.sample.AvroWordCount$Map.map(AvroWordCount.java:1)
>>>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>>> at org.apache.hadoop.mrunit.mapreduce.MapDriver.run(MapDriver.java:221)
>>>> at
>>>> amazon.fclm.labormetrics.mapreduce.sample.AvroWordCountTest.TestMrUnitMapper(AvroWordCountTest.java:61)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at
>>>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>>>> at
>>>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>>>> at
>>>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
>>>> at
>>>> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>>>> at
>>>> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>>>> at
>>>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
>>>> at
>>>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>>>> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
>>>> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
>>>> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
>>>> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
>>>> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
>>>> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
>>>> at
>>>> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
>>>> at
>>>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>>>> at
>>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
>>>> at
>>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
>>>> at
>>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
>>>> at
>>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>>>>
>>>> Any help will be appreciated. Thanks!
>>>>
>>>>
>>>
>>
>
Re: Avro + MRUnit
Posted by Tanya Bansal <ta...@gmail.com>.
Has anyone else faced the same problem?
On Tue, Dec 11, 2012 at 10:38 AM, Tanya Bansal <ta...@gmail.com>wrote:
> Thanks for the quick response.
>
> Here is the source code : http://pastebin.com/6t12ZZ5T
>
> I have created the Count class on my own.
>
> Thanks
> -Tanya
>
>
>
>
>
>
> On Tue, Dec 11, 2012 at 1:58 AM, Dave Beech <da...@paraliatech.com> wrote:
>
>> Hi Tanya
>>
>> How was your Count class created? Did you write it yourself or did you
>> use Avro tools to generate code from a schema? Something is mixed up
>> because Avro is trying to write a "generic" record whereas I'd expect it to
>> be using "specific" records in your case.
>>
>> It's hard to work out the problem from just the stacktrace alone, but if
>> you could share your code (using github gists or pastebin.com), I'd be
>> happy to take a look.
>>
>> Cheers
>> Dave
>>
>>
>> On 11 December 2012 03:53, Tanya Bansal <ta...@gmail.com> wrote:
>>
>>> Are there any examples out there using Apache and MRUnit?
>>>
>>> I have configured the AvroSerialization class using the
>>> io.serializations string
>>> I am getting the following exception when I call mapDriver.run():
>>>
>>> java.lang.ClassCastException:
>>> amazon.fclm.labormetrics.mapreduce.sample.AvroWordCount$Count cannot be
>>> cast to org.apache.avro.generic.IndexedRecord
>>> at org.apache.avro.generic.GenericData.getField(GenericData.java:518)
>>> at org.apache.avro.generic.GenericData.getField(GenericData.java:533)
>>> at
>>> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:103)
>>> at
>>> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
>>> at
>>> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:57)
>>> at
>>> org.apache.avro.hadoop.io.AvroSerializer.serialize(AvroSerializer.java:103)
>>> at
>>> org.apache.avro.hadoop.io.AvroSerializer.serialize(AvroSerializer.java:45)
>>> at org.apache.hadoop.mrunit.Serialization.copy(Serialization.java:62)
>>> at org.apache.hadoop.mrunit.Serialization.copy(Serialization.java:81)
>>> at
>>> org.apache.hadoop.mrunit.mapreduce.mock.MockContextWrapper$4.answer(MockContextWrapper.java:78)
>>> at
>>> org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
>>> at org.mockito.internal.MockHandler.handle(MockHandler.java:99)
>>> at
>>> org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36)
>>> at
>>> org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48)
>>> at
>>> org.apache.hadoop.mapreduce.Mapper$Context$$EnhancerByMockitoWithCGLIB$$4ae1b65a.write(<generated>)
>>> at
>>> amazon.fclm.labormetrics.mapreduce.sample.AvroWordCount$Map.map(AvroWordCount.java:76)
>>> at
>>> amazon.fclm.labormetrics.mapreduce.sample.AvroWordCount$Map.map(AvroWordCount.java:1)
>>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>> at org.apache.hadoop.mrunit.mapreduce.MapDriver.run(MapDriver.java:221)
>>> at
>>> amazon.fclm.labormetrics.mapreduce.sample.AvroWordCountTest.TestMrUnitMapper(AvroWordCountTest.java:61)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at
>>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>>> at
>>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>>> at
>>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
>>> at
>>> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>>> at
>>> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>>> at
>>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
>>> at
>>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>>> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
>>> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
>>> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
>>> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
>>> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
>>> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
>>> at
>>> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
>>> at
>>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>>> at
>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
>>> at
>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
>>> at
>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
>>> at
>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>>>
>>> Any help will be appreciated. Thanks!
>>>
>>>
>>
>
Re: Avro + MRUnit
Posted by Tanya Bansal <ta...@gmail.com>.
Thanks for the quick response.
Here is the source code : http://pastebin.com/6t12ZZ5T
I have created the Count class on my own.
Thanks
-Tanya
On Tue, Dec 11, 2012 at 1:58 AM, Dave Beech <da...@paraliatech.com> wrote:
> Hi Tanya
>
> How was your Count class created? Did you write it yourself or did you use
> Avro tools to generate code from a schema? Something is mixed up because
> Avro is trying to write a "generic" record whereas I'd expect it to be
> using "specific" records in your case.
>
> It's hard to work out the problem from just the stacktrace alone, but if
> you could share your code (using github gists or pastebin.com), I'd be
> happy to take a look.
>
> Cheers
> Dave
>
>
> On 11 December 2012 03:53, Tanya Bansal <ta...@gmail.com> wrote:
>
>> Are there any examples out there using Apache and MRUnit?
>>
>> I have configured the AvroSerialization class using the io.serializations
>> string
>> I am getting the following exception when I call mapDriver.run():
>>
>> java.lang.ClassCastException:
>> amazon.fclm.labormetrics.mapreduce.sample.AvroWordCount$Count cannot be
>> cast to org.apache.avro.generic.IndexedRecord
>> at org.apache.avro.generic.GenericData.getField(GenericData.java:518)
>> at org.apache.avro.generic.GenericData.getField(GenericData.java:533)
>> at
>> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:103)
>> at
>> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
>> at
>> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:57)
>> at
>> org.apache.avro.hadoop.io.AvroSerializer.serialize(AvroSerializer.java:103)
>> at
>> org.apache.avro.hadoop.io.AvroSerializer.serialize(AvroSerializer.java:45)
>> at org.apache.hadoop.mrunit.Serialization.copy(Serialization.java:62)
>> at org.apache.hadoop.mrunit.Serialization.copy(Serialization.java:81)
>> at
>> org.apache.hadoop.mrunit.mapreduce.mock.MockContextWrapper$4.answer(MockContextWrapper.java:78)
>> at
>> org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
>> at org.mockito.internal.MockHandler.handle(MockHandler.java:99)
>> at
>> org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36)
>> at
>> org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48)
>> at
>> org.apache.hadoop.mapreduce.Mapper$Context$$EnhancerByMockitoWithCGLIB$$4ae1b65a.write(<generated>)
>> at
>> amazon.fclm.labormetrics.mapreduce.sample.AvroWordCount$Map.map(AvroWordCount.java:76)
>> at
>> amazon.fclm.labormetrics.mapreduce.sample.AvroWordCount$Map.map(AvroWordCount.java:1)
>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>> at org.apache.hadoop.mrunit.mapreduce.MapDriver.run(MapDriver.java:221)
>> at
>> amazon.fclm.labormetrics.mapreduce.sample.AvroWordCountTest.TestMrUnitMapper(AvroWordCountTest.java:61)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>> at
>> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>> at
>> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
>> at
>> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>> at
>> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>> at
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
>> at
>> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
>> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
>> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
>> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
>> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
>> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
>> at
>> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
>> at
>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>>
>> Any help will be appreciated. Thanks!
>>
>>
>
Re: Avro + MRUnit
Posted by Dave Beech <da...@paraliatech.com>.
Hi Tanya
How was your Count class created? Did you write it yourself or did you use
Avro tools to generate code from a schema? Something is mixed up because
Avro is trying to write a "generic" record whereas I'd expect it to be
using "specific" records in your case.
It's hard to work out the problem from just the stacktrace alone, but if
you could share your code (using github gists or pastebin.com), I'd be
happy to take a look.
Cheers
Dave
On 11 December 2012 03:53, Tanya Bansal <ta...@gmail.com> wrote:
> Are there any examples out there using Apache and MRUnit?
>
> I have configured the AvroSerialization class using the io.serializations
> string
> I am getting the following exception when I call mapDriver.run():
>
> java.lang.ClassCastException:
> amazon.fclm.labormetrics.mapreduce.sample.AvroWordCount$Count cannot be
> cast to org.apache.avro.generic.IndexedRecord
> at org.apache.avro.generic.GenericData.getField(GenericData.java:518)
> at org.apache.avro.generic.GenericData.getField(GenericData.java:533)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:103)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:57)
> at
> org.apache.avro.hadoop.io.AvroSerializer.serialize(AvroSerializer.java:103)
> at
> org.apache.avro.hadoop.io.AvroSerializer.serialize(AvroSerializer.java:45)
> at org.apache.hadoop.mrunit.Serialization.copy(Serialization.java:62)
> at org.apache.hadoop.mrunit.Serialization.copy(Serialization.java:81)
> at
> org.apache.hadoop.mrunit.mapreduce.mock.MockContextWrapper$4.answer(MockContextWrapper.java:78)
> at
> org.mockito.internal.stubbing.StubbedInvocationMatcher.answer(StubbedInvocationMatcher.java:34)
> at org.mockito.internal.MockHandler.handle(MockHandler.java:99)
> at
> org.mockito.internal.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:36)
> at
> org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:48)
> at
> org.apache.hadoop.mapreduce.Mapper$Context$$EnhancerByMockitoWithCGLIB$$4ae1b65a.write(<generated>)
> at
> amazon.fclm.labormetrics.mapreduce.sample.AvroWordCount$Map.map(AvroWordCount.java:76)
> at
> amazon.fclm.labormetrics.mapreduce.sample.AvroWordCount$Map.map(AvroWordCount.java:1)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mrunit.mapreduce.MapDriver.run(MapDriver.java:221)
> at
> amazon.fclm.labormetrics.mapreduce.sample.AvroWordCountTest.TestMrUnitMapper(AvroWordCountTest.java:61)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
>
> Any help will be appreciated. Thanks!
>
>