You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mrunit.apache.org by Tom Wheeler <tw...@cloudera.com> on 2012/10/19 17:52:36 UTC
NoSuchMethodError using withCounters method (MRUnit 0.9.0)
I saw in the MRUnit blog that the 0.9.0 release contains an improved
method of checking counter values. I've been trying to use it, but
executing my updated test throws a NoSuchMethodError at runtime.
Since withCounter is a relatively new method, there aren't a lot of
examples yet, and it's possible I'm using it incorrectly. If not,
then I'll file a JIRA.
My Mapper increments a counter corresponding to the level field in a
line from a Log4J log file. I am confident that the mapper works OK,
as does the original version of my test which uses a JUnit assertion
to verify the counter value:
@Test
public void testWithWarnEventOriginal() {
LongWritable key = new LongWritable();
Text value = new Text("2012-09-06 22:16:49.391 CDT WARN \"My Message\"");
mapDriver.withInput(key, value);
mapDriver.runTest();
assertEquals(1, mapDriver.getCounters().findCounter(Level.WARN).getValue());
}
My updated version of the test simply replaces the assertEquals
following the runTest method with a call to withCounters that precedes
runTest.
@Test
public void testWithWarnEventUpdated() {
LongWritable key = new LongWritable();
Text value = new Text("2012-09-06 22:16:49.391 CDT WARN \"My Message\"");
mapDriver.withInput(key, value);
mapDriver.withCounter(Level.WARN, 1);
mapDriver.runTest();
}
However, executing this test throws an error at runtime:
java.lang.NoSuchMethodError:
org.apache.hadoop.mapred.Counters.findCounter(Ljava/lang/Enum;)Lorg/apache/hadoop/mapred/Counters$Counter;
at org.apache.hadoop.mrunit.counters.CounterWrapper.findCounterValue(CounterWrapper.java:64)
at org.apache.hadoop.mrunit.TestDriver.validate(TestDriver.java:368)
at org.apache.hadoop.mrunit.MapDriverBase.runTest(MapDriverBase.java:152)
at org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:137)
at logevent.v4.MapperTestCase.testWithWarnEventUpdated(MapperTestCase.java:40)
... excess stacktrace removed for brevity ...
Is it a bug in my test or could it be a bug in MRUnit?
Re: NoSuchMethodError using withCounters method (MRUnit 0.9.0)
Posted by Brock Noland <br...@cloudera.com>.
Awesome great to know!
On Fri, Oct 19, 2012 at 11:26 AM, Tom Wheeler <tw...@cloudera.com> wrote:
> On Fri, Oct 19, 2012 at 11:15 AM, Brock Noland <br...@cloudera.com> wrote:
> > Out of curiosity, would you mind trying the hadoop2 version? I know they
> > made some internal changes to mr1.
>
> That fixed it -- thanks, Brock!
>
> I had originally written this for CDH3 and didn't think to use the
> hadoop2 version of MRUnit when I upgraded my dev environment to CDH4
> because all the other tests happened to work.
>
> Thanks again,
>
> Tom
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: NoSuchMethodError using withCounters method (MRUnit 0.9.0)
Posted by Tom Wheeler <tw...@cloudera.com>.
On Fri, Oct 19, 2012 at 11:15 AM, Brock Noland <br...@cloudera.com> wrote:
> Out of curiosity, would you mind trying the hadoop2 version? I know they
> made some internal changes to mr1.
That fixed it -- thanks, Brock!
I had originally written this for CDH3 and didn't think to use the
hadoop2 version of MRUnit when I upgraded my dev environment to CDH4
because all the other tests happened to work.
Thanks again,
Tom
Re: NoSuchMethodError using withCounters method (MRUnit 0.9.0)
Posted by Brock Noland <br...@cloudera.com>.
Hi,
Out of curiosity, would you mind trying the hadoop2 version? I know they
made some internal changes to mr1.
Brock
On Fri, Oct 19, 2012 at 11:13 AM, Tom Wheeler <tw...@cloudera.com> wrote:
> On Fri, Oct 19, 2012 at 11:05 AM, Bertrand Dechoux <de...@gmail.com>
> wrote:
> > Just out of curiosity what is your exact version of MRUnit? There is a
> > classifier (1 vs 2).
> > And what is the version of hadoop?
> >
> > My first reaction would be that this exact method does not exist in the
> > provided implementation of hadoop (which sounds strange but..)
>
> Hi Bertrand,
>
> Thanks for your help. The JAR file in my classpath is
> mrunit-0.9.0-incubating-hadoop1.jar.
>
> I'm using the MRv1 version of Hadoop that's part of CDH4; specifically
> "Hadoop 2.0.0-cdh4.0.1" as returned by the "hadoop version" command.
> The MapDriver referenced in my test is imported as
> org.apache.hadoop.mrunit.MapDriver.
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: NoSuchMethodError using withCounters method (MRUnit 0.9.0)
Posted by Tom Wheeler <tw...@cloudera.com>.
On Fri, Oct 19, 2012 at 11:05 AM, Bertrand Dechoux <de...@gmail.com> wrote:
> Just out of curiosity what is your exact version of MRUnit? There is a
> classifier (1 vs 2).
> And what is the version of hadoop?
>
> My first reaction would be that this exact method does not exist in the
> provided implementation of hadoop (which sounds strange but..)
Hi Bertrand,
Thanks for your help. The JAR file in my classpath is
mrunit-0.9.0-incubating-hadoop1.jar.
I'm using the MRv1 version of Hadoop that's part of CDH4; specifically
"Hadoop 2.0.0-cdh4.0.1" as returned by the "hadoop version" command.
The MapDriver referenced in my test is imported as
org.apache.hadoop.mrunit.MapDriver.
Re: NoSuchMethodError using withCounters method (MRUnit 0.9.0)
Posted by Bertrand Dechoux <de...@gmail.com>.
Just out of curiosity what is your exact version of MRUnit? There is a
classifier (1 vs 2).
And what is the version of hadoop?
My first reaction would be that this exact method does not exist in the
provided implementation of hadoop (which sounds strange but..)
Regards
Bertrand
On Fri, Oct 19, 2012 at 5:52 PM, Tom Wheeler <tw...@cloudera.com> wrote:
> I saw in the MRUnit blog that the 0.9.0 release contains an improved
> method of checking counter values. I've been trying to use it, but
> executing my updated test throws a NoSuchMethodError at runtime.
> Since withCounter is a relatively new method, there aren't a lot of
> examples yet, and it's possible I'm using it incorrectly. If not,
> then I'll file a JIRA.
>
> My Mapper increments a counter corresponding to the level field in a
> line from a Log4J log file. I am confident that the mapper works OK,
> as does the original version of my test which uses a JUnit assertion
> to verify the counter value:
>
> @Test
> public void testWithWarnEventOriginal() {
> LongWritable key = new LongWritable();
> Text value = new Text("2012-09-06 22:16:49.391 CDT WARN \"My
> Message\"");
>
> mapDriver.withInput(key, value);
> mapDriver.runTest();
> assertEquals(1,
> mapDriver.getCounters().findCounter(Level.WARN).getValue());
> }
>
> My updated version of the test simply replaces the assertEquals
> following the runTest method with a call to withCounters that precedes
> runTest.
>
> @Test
> public void testWithWarnEventUpdated() {
> LongWritable key = new LongWritable();
> Text value = new Text("2012-09-06 22:16:49.391 CDT WARN \"My
> Message\"");
>
> mapDriver.withInput(key, value);
> mapDriver.withCounter(Level.WARN, 1);
> mapDriver.runTest();
> }
>
> However, executing this test throws an error at runtime:
>
> java.lang.NoSuchMethodError:
>
> org.apache.hadoop.mapred.Counters.findCounter(Ljava/lang/Enum;)Lorg/apache/hadoop/mapred/Counters$Counter;
> at
> org.apache.hadoop.mrunit.counters.CounterWrapper.findCounterValue(CounterWrapper.java:64)
> at
> org.apache.hadoop.mrunit.TestDriver.validate(TestDriver.java:368)
> at
> org.apache.hadoop.mrunit.MapDriverBase.runTest(MapDriverBase.java:152)
> at org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:137)
> at
> logevent.v4.MapperTestCase.testWithWarnEventUpdated(MapperTestCase.java:40)
> ... excess stacktrace removed for brevity ...
>
> Is it a bug in my test or could it be a bug in MRUnit?
>
--
Bertrand Dechoux