You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Jason Venner <ja...@gmail.com> on 2009/08/26 16:29:49 UTC
Re: Testing Hadoop job
I put together a framework for the Pro Hadoop book that I use quite a bit,
and has some documentation in the book examples ;)
I haven't tried it with 0.20.0 however.
The nicest thing that I did with the framework was provide a way to run a
persistent mini virtual cluster for running multiple tests on.
On Wed, Aug 26, 2009 at 4:50 PM, Nikhil Sawant <ns...@yahoo-inc.com>wrote:
> hi
>
> can u guys suggest some hadoop unit testing framework apart from MRUnit???
> i have used MRUnit but i m not sure abt its feasibilty and support to
> hadoop 0.20.....
> i could not find a proper documentation for MRUnit, is it available
> anywhere?
>
> --
> cheers
> nikhil
>
>
--
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals
Re: Testing Hadoop job
Posted by Jason Venner <ja...@gmail.com>.
Ahh, the pro hadoop stuff handles arbitrary input and output formats and
partitioners.
On Fri, Aug 28, 2009 at 5:50 PM, Aaron Kimball <aa...@cloudera.com> wrote:
> Hi Nikhil,
>
> MRUnit now supports the 0.20 API as of
> https://issues.apache.org/jira/browse/MAPREDUCE-800. There are no plans to
> involve partitioners in MRUnit; it is for mappers and reducers only, and
> not
> for full jobs involving input/output formats, partitioners, etc. Use the
> LocalJobRunner for that.
>
> See www.cloudera.com/hadoop-mrunit for more (slightly out-of-date)
> information.
> - Aaron
>
> On Fri, Aug 28, 2009 at 9:15 AM, Nikhil Sawant <nsawant@yahoo-inc.com
> >wrote:
>
> > hi
> > thanks Jason, for prompt reply i will go through "pro hadoop". already on
> > my to-do list
> >
> > any idea abt the MRUnit??? has anyone used it?
> > i think it is useful as it allows dummy map and reduce drivers which
> > accepts (K,V) pair/s and checks the o/p (K,V) pair/s with the expected
> > (K,V)......it gives gr8 debugging capabilities while implementing complex
> > logics. (all "System.outs" can be seen on console itself!!)
> > i have used the basic functionalities of MRUnit testing framework
> > i would like to know the limitations (e.g. i found out that MRUnit does
> not
> > check the partioner logic) and its feasibility with hadoop 0.20...
> > No proper documentation i found ! :(
> >
> > cheers
> > nikhil
> >
> >
> > Jason Venner wrote:
> >
> >> I put together a framework for the Pro Hadoop book that I use quite a
> bit,
> >> and has some documentation in the book examples ;)
> >> I haven't tried it with 0.20.0 however.
> >>
> >> The nicest thing that I did with the framework was provide a way to run
> a
> >> persistent mini virtual cluster for running multiple tests on.
> >>
> >> On Wed, Aug 26, 2009 at 4:50 PM, Nikhil Sawant <nsawant@yahoo-inc.com
> >> >wrote:
> >>
> >>
> >>
> >>> hi
> >>>
> >>> can u guys suggest some hadoop unit testing framework apart from
> >>> MRUnit???
> >>> i have used MRUnit but i m not sure abt its feasibilty and support to
> >>> hadoop 0.20.....
> >>> i could not find a proper documentation for MRUnit, is it available
> >>> anywhere?
> >>>
> >>> --
> >>> cheers
> >>> nikhil
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >>
> >
>
--
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals
Re: Testing Hadoop job
Posted by Aaron Kimball <aa...@cloudera.com>.
Hi Nikhil,
MRUnit now supports the 0.20 API as of
https://issues.apache.org/jira/browse/MAPREDUCE-800. There are no plans to
involve partitioners in MRUnit; it is for mappers and reducers only, and not
for full jobs involving input/output formats, partitioners, etc. Use the
LocalJobRunner for that.
See www.cloudera.com/hadoop-mrunit for more (slightly out-of-date)
information.
- Aaron
On Fri, Aug 28, 2009 at 9:15 AM, Nikhil Sawant <ns...@yahoo-inc.com>wrote:
> hi
> thanks Jason, for prompt reply i will go through "pro hadoop". already on
> my to-do list
>
> any idea abt the MRUnit??? has anyone used it?
> i think it is useful as it allows dummy map and reduce drivers which
> accepts (K,V) pair/s and checks the o/p (K,V) pair/s with the expected
> (K,V)......it gives gr8 debugging capabilities while implementing complex
> logics. (all "System.outs" can be seen on console itself!!)
> i have used the basic functionalities of MRUnit testing framework
> i would like to know the limitations (e.g. i found out that MRUnit does not
> check the partioner logic) and its feasibility with hadoop 0.20...
> No proper documentation i found ! :(
>
> cheers
> nikhil
>
>
> Jason Venner wrote:
>
>> I put together a framework for the Pro Hadoop book that I use quite a bit,
>> and has some documentation in the book examples ;)
>> I haven't tried it with 0.20.0 however.
>>
>> The nicest thing that I did with the framework was provide a way to run a
>> persistent mini virtual cluster for running multiple tests on.
>>
>> On Wed, Aug 26, 2009 at 4:50 PM, Nikhil Sawant <nsawant@yahoo-inc.com
>> >wrote:
>>
>>
>>
>>> hi
>>>
>>> can u guys suggest some hadoop unit testing framework apart from
>>> MRUnit???
>>> i have used MRUnit but i m not sure abt its feasibilty and support to
>>> hadoop 0.20.....
>>> i could not find a proper documentation for MRUnit, is it available
>>> anywhere?
>>>
>>> --
>>> cheers
>>> nikhil
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>
Re: Testing Hadoop job
Posted by Nikhil Sawant <ns...@yahoo-inc.com>.
hi
thanks Jason, for prompt reply i will go through "pro hadoop". already
on my to-do list
any idea abt the MRUnit??? has anyone used it?
i think it is useful as it allows dummy map and reduce drivers which
accepts (K,V) pair/s and checks the o/p (K,V) pair/s with the expected
(K,V)......it gives gr8 debugging capabilities while implementing
complex logics. (all "System.outs" can be seen on console itself!!)
i have used the basic functionalities of MRUnit testing framework
i would like to know the limitations (e.g. i found out that MRUnit does
not check the partioner logic) and its feasibility with hadoop 0.20...
No proper documentation i found ! :(
cheers
nikhil
Jason Venner wrote:
> I put together a framework for the Pro Hadoop book that I use quite a bit,
> and has some documentation in the book examples ;)
> I haven't tried it with 0.20.0 however.
>
> The nicest thing that I did with the framework was provide a way to run a
> persistent mini virtual cluster for running multiple tests on.
>
> On Wed, Aug 26, 2009 at 4:50 PM, Nikhil Sawant <ns...@yahoo-inc.com>wrote:
>
>
>> hi
>>
>> can u guys suggest some hadoop unit testing framework apart from MRUnit???
>> i have used MRUnit but i m not sure abt its feasibilty and support to
>> hadoop 0.20.....
>> i could not find a proper documentation for MRUnit, is it available
>> anywhere?
>>
>> --
>> cheers
>> nikhil
>>
>>
>>
>
>
>