You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mrunit.apache.org by Bertrand Dechoux <de...@gmail.com> on 2012/06/14 10:12:29 UTC

Checking that no counter was incremented

Hello,

I would like to check that no counter was incremented. I was just wondering
how you would do it.
After reflection, I guess the hard part is that Hadoop itself uses the
counters so I would need to separate my counters from the 'standard' ones.

My use case is however simple. I have a Mapper/Reducer which might
increment a few counters on certain 'error' cases and I would like to
verify a good case for which no counters should have been incremented.
The issue is that I might no know the name of the counters because they are
dynamic. Even the group name is kind of dynamic because for warning
counters, I use the class name and so it might change after refactoring.

Any hints, leads are welcome

Thanks in advance

Bertrand

Re: Checking that no counter was incremented

Posted by Bertrand Dechoux <de...@gmail.com>.
More informations for users that might have the same question.

There is a method that might be useful :
driver.withCounters(final Counters ctrs)

If someone is courageous, it could use it as an entry point for a mock or a
custom implementation.

As for a feature request, what could be done is to reverse the testing of
'expected counters'.
At the moment, TestDriver.validate(final CounterWrapper counterWrapper) is
testing that for every expected counters there is a matching
actual/observed counter.
What could be done (with a flag, in order to have both behaviour?) is to
test that for every actual/observed counters, there is a matching expected
counter.

That would be a stricter assertion, not useful everytime but useful
sometimes. (like the order of a mapper/reducer outputs).

Regards

Bertrand


On Mon, Jun 18, 2012 at 10:54 AM, Bertrand Dechoux <de...@gmail.com>wrote:

> Thanks for the answer. I obviously wasn't looking for a hard commitment,
> merely opening the discussion about it. But at least your clear.
> I might have a look at it if I have enough time because it matters do me.
> But first I will have to learn more about MRunit.
>
> Have a good week,
>
> Bertrand
>
>
> On Mon, Jun 18, 2012 at 10:05 AM, Dave Beech <db...@apache.org> wrote:
>
>> Hi Bertrand
>>
>> I have created issues MRUNIT-119 and MRUNIT-120 on JIRA to cover your
>> feature requests. Unfortunately I can't give you any real idea of if/when
>> they will be implemented, but we will discuss and keep the issues updated
>> with comments etc.
>>
>> Regards,
>> Dave
>>
>>
>> On 14 June 2012 09:12, Bertrand Dechoux <de...@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I would like to check that no counter was incremented. I was just
>>> wondering how you would do it.
>>> After reflection, I guess the hard part is that Hadoop itself uses the
>>> counters so I would need to separate my counters from the 'standard' ones.
>>>
>>> My use case is however simple. I have a Mapper/Reducer which might
>>> increment a few counters on certain 'error' cases and I would like to
>>> verify a good case for which no counters should have been incremented.
>>> The issue is that I might no know the name of the counters because they
>>> are dynamic. Even the group name is kind of dynamic because for warning
>>> counters, I use the class name and so it might change after refactoring.
>>>
>>> Any hints, leads are welcome
>>>
>>> Thanks in advance
>>>
>>> Bertrand
>>>
>>
>>
>
>
> --
> Bertrand Dechoux
>



-- 
Bertrand Dechoux

Re: Checking that no counter was incremented

Posted by Bertrand Dechoux <de...@gmail.com>.
Thanks for the answer. I obviously wasn't looking for a hard commitment,
merely opening the discussion about it. But at least your clear.
I might have a look at it if I have enough time because it matters do me.
But first I will have to learn more about MRunit.

Have a good week,

Bertrand

On Mon, Jun 18, 2012 at 10:05 AM, Dave Beech <db...@apache.org> wrote:

> Hi Bertrand
>
> I have created issues MRUNIT-119 and MRUNIT-120 on JIRA to cover your
> feature requests. Unfortunately I can't give you any real idea of if/when
> they will be implemented, but we will discuss and keep the issues updated
> with comments etc.
>
> Regards,
> Dave
>
>
> On 14 June 2012 09:12, Bertrand Dechoux <de...@gmail.com> wrote:
>
>> Hello,
>>
>> I would like to check that no counter was incremented. I was just
>> wondering how you would do it.
>> After reflection, I guess the hard part is that Hadoop itself uses the
>> counters so I would need to separate my counters from the 'standard' ones.
>>
>> My use case is however simple. I have a Mapper/Reducer which might
>> increment a few counters on certain 'error' cases and I would like to
>> verify a good case for which no counters should have been incremented.
>> The issue is that I might no know the name of the counters because they
>> are dynamic. Even the group name is kind of dynamic because for warning
>> counters, I use the class name and so it might change after refactoring.
>>
>> Any hints, leads are welcome
>>
>> Thanks in advance
>>
>> Bertrand
>>
>
>


-- 
Bertrand Dechoux

Re: Checking that no counter was incremented

Posted by Dave Beech <db...@apache.org>.
Hi Bertrand

I have created issues MRUNIT-119 and MRUNIT-120 on JIRA to cover your
feature requests. Unfortunately I can't give you any real idea of if/when
they will be implemented, but we will discuss and keep the issues updated
with comments etc.

Regards,
Dave

On 14 June 2012 09:12, Bertrand Dechoux <de...@gmail.com> wrote:

> Hello,
>
> I would like to check that no counter was incremented. I was just
> wondering how you would do it.
> After reflection, I guess the hard part is that Hadoop itself uses the
> counters so I would need to separate my counters from the 'standard' ones.
>
> My use case is however simple. I have a Mapper/Reducer which might
> increment a few counters on certain 'error' cases and I would like to
> verify a good case for which no counters should have been incremented.
> The issue is that I might no know the name of the counters because they
> are dynamic. Even the group name is kind of dynamic because for warning
> counters, I use the class name and so it might change after refactoring.
>
> Any hints, leads are welcome
>
> Thanks in advance
>
> Bertrand
>