You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Edson Ramiro <er...@gmail.com> on 2011/02/02 14:44:10 UTC
MRUnit and Herriot
Hi all,
Plz, could you explain me the difference between MRUnit and Herriot?
I've read the documentation of both and they seem very similar to me.
Is Herriot an evolution of MRUnit?
What can Herriot do that MRUnit can't?
Thanks in Advance
--
Edson Ramiro Lucas Filho
{skype, twitter, gtalk}: erlfilho
http://www.inf.ufpr.br/erlf07/
Re: MRUnit and Herriot
Posted by Konstantin Boudnik <co...@apache.org>.
On Thu, Feb 10, 2011 at 08:39, Edson Ramiro <er...@gmail.com> wrote:
> Hi,
>
> I took a look around on the Internet, but I didn't find any docs about
> MiniDFS
> and MiniMRCluster. Is there docs about them?
>
> It remember me this phrase I got from the Herriot [1] page.
> "As always your best source of information and knowledge about any software
> system is its source code" :)
Yes, this still holds ;) Source code is your best friend for a number
of reasons:
- this is _the_ best documentation for the code and shows what an
application does
- it is always up-to-date
- developers can focus on their development/testing rather then
writing an end-user documents about some internals (which no-one but
other developers will ever need)
> Do you think is possible to have just one tool to cover all kinds of tests?
Sure, why not? I am also a big believer that a single OS would do just fine.
> Another question, do you know if is possible to evaluate a MR program, eg
> sort, with Herriot considering several test data?
Absolutely... Herriot does run work-loads against a physical clusters.
So, I don't see why it can be impossible. Would be most effective use
of your time? Perhaps not, because Herriot requires a specially
tailored (instrumented) cluster to be executed against.
What you need, I think, is a simple way to get a jar file containing
some tests, drop it to a cluster's gateway machine and run then. Looks
like as what we are trying to achieve in iTest I have mentioned
earlier.
Cos
> Thanks in Advance
>
> --
> Edson Ramiro Lucas Filho
> {skype, twitter, gtalk}: erlfilho
> http://www.inf.ufpr.br/erlf07/
>
>
> On Mon, Feb 7, 2011 at 10:29 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
>> On Mon, Feb 7, 2011 at 04:20, Edson Ramiro <er...@gmail.com> wrote:
>> > Well, I'm studying the Hadoop test tools to evaluate some (if there are)
>> > deficiences, also trying to compare these tools to see what one cover
>> that
>> > other doesn't and what is possible to do with each one.
>>
>> There's also a simulated test cluster infrastructure called MiniDFS
>> and MiniMRCluster to allow you to develop functional tests without
>> actual cluster deployment.
>>
>> > As far as I know we have just Herriot and MRUnit for test, and them do
>> > different things as you said me :)
>> >
>> > I'm very interested in your initial version, is there a link?
>>
>> Not at the moment, but I will send it here as soon as a initial
>> version is pushed out.
>>
>> >
>> > Thanks in advance
>> >
>> > --
>> > Edson Ramiro Lucas Filho
>> > {skype, twitter, gtalk}: erlfilho
>> > http://www.inf.ufpr.br/erlf07/
>> >
>> >
>> > On Fri, Feb 4, 2011 at 3:40 AM, Konstantin Boudnik <co...@apache.org>
>> wrote:
>> >
>> >> Yes, Herriot can be used for integration tests of MR. Unit test is a
>> very
>> >> different thing and normally is done against a 'unit of compilation'
>> e.g. a
>> >> class, etc. Typically you won't expect to do unit tests against a
>> deployed
>> >> cluster.
>> >>
>> >> There is fault injection framework wich works at the level of functional
>> >> tests
>> >> (with mini-clusters). Shortly we'll be opening an initial version of
>> smoke
>> >> and
>> >> integration test framework (maven and JUnit based).
>> >>
>> >> It'd be easier to provide you with a hint if you care to explain what
>> >> you're
>> >> trying to solve.
>> >>
>> >> Cos
>> >>
>> >> On Thu, Feb 03, 2011 at 10:25AM, Edson Ramiro wrote:
>> >> > Thank you a lot Konstantin, you cleared my mind.
>> >> >
>> >> > So, Herriot is a framework designed to test Hadoop as a whole, and
>> (IMHO)
>> >> is
>> >> > a tool for help Hadoop developers and not for who is developing MR
>> >> programs,
>> >> > but can we use Herriot to do unit, integration or other tests on our
>> MR
>> >> > jobs?
>> >> >
>> >> > Do you know another test tool or test framework for Hadoop?
>> >> >
>> >> > Thanks in Advance
>> >> >
>> >> > --
>> >> > Edson Ramiro Lucas Filho
>> >> > {skype, twitter, gtalk}: erlfilho
>> >> > http://www.inf.ufpr.br/erlf07/
>> >> >
>> >> >
>> >> > On Wed, Feb 2, 2011 at 4:58 PM, Konstantin Boudnik <co...@apache.org>
>> >> wrote:
>> >> >
>> >> > > (Moving to common-user where this belongs)
>> >> > >
>> >> > > Herriot is system test framework which runs against a real physical
>> >> > > cluster deployed with a specially crafted build of Hadoop. That
>> >> > > instrumented build of provides an extra APIs not available in Hadoop
>> >> > > otherwise. These APIs are created to facilitate cluster software
>> >> > > testability. Herriot isn't limited by MR but also covered (although
>> in
>> >> > > a somewhat lesser extend) HDFS side of Hadoop.
>> >> > >
>> >> > > MRunit is for MR job "unit" testing as in making sure that your MR
>> job
>> >> > > is ok and/or to allow you to debug it locally before scale
>> deployment.
>> >> > >
>> >> > > So, long story short - they are very different ;) Herriot can do
>> >> > > intricate fault injection and can work closely with a deployed
>> cluster
>> >> > > (say control Hadoop nodes and daemons); MRUnit is focused on MR jobs
>> >> > > testing.
>> >> > >
>> >> > > Hope it helps.
>> >> > > --
>> >> > > Take care,
>> >> > > Konstantin (Cos) Boudnik
>> >> > >
>> >> > >
>> >> > > On Wed, Feb 2, 2011 at 05:44, Edson Ramiro <er...@gmail.com>
>> wrote:
>> >> > > > Hi all,
>> >> > > >
>> >> > > > Plz, could you explain me the difference between MRUnit and
>> Herriot?
>> >> > > >
>> >> > > > I've read the documentation of both and they seem very similar to
>> me.
>> >> > > >
>> >> > > > Is Herriot an evolution of MRUnit?
>> >> > > >
>> >> > > > What can Herriot do that MRUnit can't?
>> >> > > >
>> >> > > > Thanks in Advance
>> >> > > >
>> >> > > > --
>> >> > > > Edson Ramiro Lucas Filho
>> >> > > > {skype, twitter, gtalk}: erlfilho
>> >> > > > http://www.inf.ufpr.br/erlf07/
>> >> > > >
>> >> > >
>> >>
>> >> -----BEGIN PGP SIGNATURE-----
>> >> Version: GnuPG v1.4.10 (GNU/Linux)
>> >>
>> >> iF4EAREIAAYFAk1LkUYACgkQenyFlstYjhIyYwD9HM7YvfdcvBuqdN24No5T4dLe
>> >> lDLVlnEs8QIN4V7RqAYBAJ8liUG2YZ+c/wvWL3/lVAGY+Fqls0k4OYLG4rXJrwwD
>> >> =h/66
>> >> -----END PGP SIGNATURE-----
>> >>
>> >>
>> >
>>
>
Re: MRUnit and Herriot
Posted by Edson Ramiro <er...@gmail.com>.
Hi,
I took a look around on the Internet, but I didn't find any docs about
MiniDFS
and MiniMRCluster. Is there docs about them?
It remember me this phrase I got from the Herriot [1] page.
"As always your best source of information and knowledge about any software
system is its source code" :)
Do you think is possible to have just one tool to cover all kinds of tests?
Another question, do you know if is possible to evaluate a MR program, eg
sort,
with Herriot considering several test data?
Thanks in Advance
--
Edson Ramiro Lucas Filho
{skype, twitter, gtalk}: erlfilho
http://www.inf.ufpr.br/erlf07/
On Mon, Feb 7, 2011 at 10:29 PM, Konstantin Boudnik <co...@apache.org> wrote:
> On Mon, Feb 7, 2011 at 04:20, Edson Ramiro <er...@gmail.com> wrote:
> > Well, I'm studying the Hadoop test tools to evaluate some (if there are)
> > deficiences, also trying to compare these tools to see what one cover
> that
> > other doesn't and what is possible to do with each one.
>
> There's also a simulated test cluster infrastructure called MiniDFS
> and MiniMRCluster to allow you to develop functional tests without
> actual cluster deployment.
>
> > As far as I know we have just Herriot and MRUnit for test, and them do
> > different things as you said me :)
> >
> > I'm very interested in your initial version, is there a link?
>
> Not at the moment, but I will send it here as soon as a initial
> version is pushed out.
>
> >
> > Thanks in advance
> >
> > --
> > Edson Ramiro Lucas Filho
> > {skype, twitter, gtalk}: erlfilho
> > http://www.inf.ufpr.br/erlf07/
> >
> >
> > On Fri, Feb 4, 2011 at 3:40 AM, Konstantin Boudnik <co...@apache.org>
> wrote:
> >
> >> Yes, Herriot can be used for integration tests of MR. Unit test is a
> very
> >> different thing and normally is done against a 'unit of compilation'
> e.g. a
> >> class, etc. Typically you won't expect to do unit tests against a
> deployed
> >> cluster.
> >>
> >> There is fault injection framework wich works at the level of functional
> >> tests
> >> (with mini-clusters). Shortly we'll be opening an initial version of
> smoke
> >> and
> >> integration test framework (maven and JUnit based).
> >>
> >> It'd be easier to provide you with a hint if you care to explain what
> >> you're
> >> trying to solve.
> >>
> >> Cos
> >>
> >> On Thu, Feb 03, 2011 at 10:25AM, Edson Ramiro wrote:
> >> > Thank you a lot Konstantin, you cleared my mind.
> >> >
> >> > So, Herriot is a framework designed to test Hadoop as a whole, and
> (IMHO)
> >> is
> >> > a tool for help Hadoop developers and not for who is developing MR
> >> programs,
> >> > but can we use Herriot to do unit, integration or other tests on our
> MR
> >> > jobs?
> >> >
> >> > Do you know another test tool or test framework for Hadoop?
> >> >
> >> > Thanks in Advance
> >> >
> >> > --
> >> > Edson Ramiro Lucas Filho
> >> > {skype, twitter, gtalk}: erlfilho
> >> > http://www.inf.ufpr.br/erlf07/
> >> >
> >> >
> >> > On Wed, Feb 2, 2011 at 4:58 PM, Konstantin Boudnik <co...@apache.org>
> >> wrote:
> >> >
> >> > > (Moving to common-user where this belongs)
> >> > >
> >> > > Herriot is system test framework which runs against a real physical
> >> > > cluster deployed with a specially crafted build of Hadoop. That
> >> > > instrumented build of provides an extra APIs not available in Hadoop
> >> > > otherwise. These APIs are created to facilitate cluster software
> >> > > testability. Herriot isn't limited by MR but also covered (although
> in
> >> > > a somewhat lesser extend) HDFS side of Hadoop.
> >> > >
> >> > > MRunit is for MR job "unit" testing as in making sure that your MR
> job
> >> > > is ok and/or to allow you to debug it locally before scale
> deployment.
> >> > >
> >> > > So, long story short - they are very different ;) Herriot can do
> >> > > intricate fault injection and can work closely with a deployed
> cluster
> >> > > (say control Hadoop nodes and daemons); MRUnit is focused on MR jobs
> >> > > testing.
> >> > >
> >> > > Hope it helps.
> >> > > --
> >> > > Take care,
> >> > > Konstantin (Cos) Boudnik
> >> > >
> >> > >
> >> > > On Wed, Feb 2, 2011 at 05:44, Edson Ramiro <er...@gmail.com>
> wrote:
> >> > > > Hi all,
> >> > > >
> >> > > > Plz, could you explain me the difference between MRUnit and
> Herriot?
> >> > > >
> >> > > > I've read the documentation of both and they seem very similar to
> me.
> >> > > >
> >> > > > Is Herriot an evolution of MRUnit?
> >> > > >
> >> > > > What can Herriot do that MRUnit can't?
> >> > > >
> >> > > > Thanks in Advance
> >> > > >
> >> > > > --
> >> > > > Edson Ramiro Lucas Filho
> >> > > > {skype, twitter, gtalk}: erlfilho
> >> > > > http://www.inf.ufpr.br/erlf07/
> >> > > >
> >> > >
> >>
> >> -----BEGIN PGP SIGNATURE-----
> >> Version: GnuPG v1.4.10 (GNU/Linux)
> >>
> >> iF4EAREIAAYFAk1LkUYACgkQenyFlstYjhIyYwD9HM7YvfdcvBuqdN24No5T4dLe
> >> lDLVlnEs8QIN4V7RqAYBAJ8liUG2YZ+c/wvWL3/lVAGY+Fqls0k4OYLG4rXJrwwD
> >> =h/66
> >> -----END PGP SIGNATURE-----
> >>
> >>
> >
>
Re: MRUnit and Herriot
Posted by Konstantin Boudnik <co...@apache.org>.
On Mon, Feb 7, 2011 at 04:20, Edson Ramiro <er...@gmail.com> wrote:
> Well, I'm studying the Hadoop test tools to evaluate some (if there are)
> deficiences, also trying to compare these tools to see what one cover that
> other doesn't and what is possible to do with each one.
There's also a simulated test cluster infrastructure called MiniDFS
and MiniMRCluster to allow you to develop functional tests without
actual cluster deployment.
> As far as I know we have just Herriot and MRUnit for test, and them do
> different things as you said me :)
>
> I'm very interested in your initial version, is there a link?
Not at the moment, but I will send it here as soon as a initial
version is pushed out.
>
> Thanks in advance
>
> --
> Edson Ramiro Lucas Filho
> {skype, twitter, gtalk}: erlfilho
> http://www.inf.ufpr.br/erlf07/
>
>
> On Fri, Feb 4, 2011 at 3:40 AM, Konstantin Boudnik <co...@apache.org> wrote:
>
>> Yes, Herriot can be used for integration tests of MR. Unit test is a very
>> different thing and normally is done against a 'unit of compilation' e.g. a
>> class, etc. Typically you won't expect to do unit tests against a deployed
>> cluster.
>>
>> There is fault injection framework wich works at the level of functional
>> tests
>> (with mini-clusters). Shortly we'll be opening an initial version of smoke
>> and
>> integration test framework (maven and JUnit based).
>>
>> It'd be easier to provide you with a hint if you care to explain what
>> you're
>> trying to solve.
>>
>> Cos
>>
>> On Thu, Feb 03, 2011 at 10:25AM, Edson Ramiro wrote:
>> > Thank you a lot Konstantin, you cleared my mind.
>> >
>> > So, Herriot is a framework designed to test Hadoop as a whole, and (IMHO)
>> is
>> > a tool for help Hadoop developers and not for who is developing MR
>> programs,
>> > but can we use Herriot to do unit, integration or other tests on our MR
>> > jobs?
>> >
>> > Do you know another test tool or test framework for Hadoop?
>> >
>> > Thanks in Advance
>> >
>> > --
>> > Edson Ramiro Lucas Filho
>> > {skype, twitter, gtalk}: erlfilho
>> > http://www.inf.ufpr.br/erlf07/
>> >
>> >
>> > On Wed, Feb 2, 2011 at 4:58 PM, Konstantin Boudnik <co...@apache.org>
>> wrote:
>> >
>> > > (Moving to common-user where this belongs)
>> > >
>> > > Herriot is system test framework which runs against a real physical
>> > > cluster deployed with a specially crafted build of Hadoop. That
>> > > instrumented build of provides an extra APIs not available in Hadoop
>> > > otherwise. These APIs are created to facilitate cluster software
>> > > testability. Herriot isn't limited by MR but also covered (although in
>> > > a somewhat lesser extend) HDFS side of Hadoop.
>> > >
>> > > MRunit is for MR job "unit" testing as in making sure that your MR job
>> > > is ok and/or to allow you to debug it locally before scale deployment.
>> > >
>> > > So, long story short - they are very different ;) Herriot can do
>> > > intricate fault injection and can work closely with a deployed cluster
>> > > (say control Hadoop nodes and daemons); MRUnit is focused on MR jobs
>> > > testing.
>> > >
>> > > Hope it helps.
>> > > --
>> > > Take care,
>> > > Konstantin (Cos) Boudnik
>> > >
>> > >
>> > > On Wed, Feb 2, 2011 at 05:44, Edson Ramiro <er...@gmail.com> wrote:
>> > > > Hi all,
>> > > >
>> > > > Plz, could you explain me the difference between MRUnit and Herriot?
>> > > >
>> > > > I've read the documentation of both and they seem very similar to me.
>> > > >
>> > > > Is Herriot an evolution of MRUnit?
>> > > >
>> > > > What can Herriot do that MRUnit can't?
>> > > >
>> > > > Thanks in Advance
>> > > >
>> > > > --
>> > > > Edson Ramiro Lucas Filho
>> > > > {skype, twitter, gtalk}: erlfilho
>> > > > http://www.inf.ufpr.br/erlf07/
>> > > >
>> > >
>>
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.10 (GNU/Linux)
>>
>> iF4EAREIAAYFAk1LkUYACgkQenyFlstYjhIyYwD9HM7YvfdcvBuqdN24No5T4dLe
>> lDLVlnEs8QIN4V7RqAYBAJ8liUG2YZ+c/wvWL3/lVAGY+Fqls0k4OYLG4rXJrwwD
>> =h/66
>> -----END PGP SIGNATURE-----
>>
>>
>
Re: MRUnit and Herriot
Posted by Edson Ramiro <er...@gmail.com>.
Well, I'm studying the Hadoop test tools to evaluate some (if there are)
deficiences, also trying to compare these tools to see what one cover that
other doesn't and what is possible to do with each one.
As far as I know we have just Herriot and MRUnit for test, and them do
different things as you said me :)
I'm very interested in your initial version, is there a link?
Thanks in advance
--
Edson Ramiro Lucas Filho
{skype, twitter, gtalk}: erlfilho
http://www.inf.ufpr.br/erlf07/
On Fri, Feb 4, 2011 at 3:40 AM, Konstantin Boudnik <co...@apache.org> wrote:
> Yes, Herriot can be used for integration tests of MR. Unit test is a very
> different thing and normally is done against a 'unit of compilation' e.g. a
> class, etc. Typically you won't expect to do unit tests against a deployed
> cluster.
>
> There is fault injection framework wich works at the level of functional
> tests
> (with mini-clusters). Shortly we'll be opening an initial version of smoke
> and
> integration test framework (maven and JUnit based).
>
> It'd be easier to provide you with a hint if you care to explain what
> you're
> trying to solve.
>
> Cos
>
> On Thu, Feb 03, 2011 at 10:25AM, Edson Ramiro wrote:
> > Thank you a lot Konstantin, you cleared my mind.
> >
> > So, Herriot is a framework designed to test Hadoop as a whole, and (IMHO)
> is
> > a tool for help Hadoop developers and not for who is developing MR
> programs,
> > but can we use Herriot to do unit, integration or other tests on our MR
> > jobs?
> >
> > Do you know another test tool or test framework for Hadoop?
> >
> > Thanks in Advance
> >
> > --
> > Edson Ramiro Lucas Filho
> > {skype, twitter, gtalk}: erlfilho
> > http://www.inf.ufpr.br/erlf07/
> >
> >
> > On Wed, Feb 2, 2011 at 4:58 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> >
> > > (Moving to common-user where this belongs)
> > >
> > > Herriot is system test framework which runs against a real physical
> > > cluster deployed with a specially crafted build of Hadoop. That
> > > instrumented build of provides an extra APIs not available in Hadoop
> > > otherwise. These APIs are created to facilitate cluster software
> > > testability. Herriot isn't limited by MR but also covered (although in
> > > a somewhat lesser extend) HDFS side of Hadoop.
> > >
> > > MRunit is for MR job "unit" testing as in making sure that your MR job
> > > is ok and/or to allow you to debug it locally before scale deployment.
> > >
> > > So, long story short - they are very different ;) Herriot can do
> > > intricate fault injection and can work closely with a deployed cluster
> > > (say control Hadoop nodes and daemons); MRUnit is focused on MR jobs
> > > testing.
> > >
> > > Hope it helps.
> > > --
> > > Take care,
> > > Konstantin (Cos) Boudnik
> > >
> > >
> > > On Wed, Feb 2, 2011 at 05:44, Edson Ramiro <er...@gmail.com> wrote:
> > > > Hi all,
> > > >
> > > > Plz, could you explain me the difference between MRUnit and Herriot?
> > > >
> > > > I've read the documentation of both and they seem very similar to me.
> > > >
> > > > Is Herriot an evolution of MRUnit?
> > > >
> > > > What can Herriot do that MRUnit can't?
> > > >
> > > > Thanks in Advance
> > > >
> > > > --
> > > > Edson Ramiro Lucas Filho
> > > > {skype, twitter, gtalk}: erlfilho
> > > > http://www.inf.ufpr.br/erlf07/
> > > >
> > >
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
>
> iF4EAREIAAYFAk1LkUYACgkQenyFlstYjhIyYwD9HM7YvfdcvBuqdN24No5T4dLe
> lDLVlnEs8QIN4V7RqAYBAJ8liUG2YZ+c/wvWL3/lVAGY+Fqls0k4OYLG4rXJrwwD
> =h/66
> -----END PGP SIGNATURE-----
>
>
Re: MRUnit and Herriot
Posted by Konstantin Boudnik <co...@apache.org>.
Yes, Herriot can be used for integration tests of MR. Unit test is a very
different thing and normally is done against a 'unit of compilation' e.g. a
class, etc. Typically you won't expect to do unit tests against a deployed
cluster.
There is fault injection framework wich works at the level of functional tests
(with mini-clusters). Shortly we'll be opening an initial version of smoke and
integration test framework (maven and JUnit based).
It'd be easier to provide you with a hint if you care to explain what you're
trying to solve.
Cos
On Thu, Feb 03, 2011 at 10:25AM, Edson Ramiro wrote:
> Thank you a lot Konstantin, you cleared my mind.
>
> So, Herriot is a framework designed to test Hadoop as a whole, and (IMHO) is
> a tool for help Hadoop developers and not for who is developing MR programs,
> but can we use Herriot to do unit, integration or other tests on our MR
> jobs?
>
> Do you know another test tool or test framework for Hadoop?
>
> Thanks in Advance
>
> --
> Edson Ramiro Lucas Filho
> {skype, twitter, gtalk}: erlfilho
> http://www.inf.ufpr.br/erlf07/
>
>
> On Wed, Feb 2, 2011 at 4:58 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
> > (Moving to common-user where this belongs)
> >
> > Herriot is system test framework which runs against a real physical
> > cluster deployed with a specially crafted build of Hadoop. That
> > instrumented build of provides an extra APIs not available in Hadoop
> > otherwise. These APIs are created to facilitate cluster software
> > testability. Herriot isn't limited by MR but also covered (although in
> > a somewhat lesser extend) HDFS side of Hadoop.
> >
> > MRunit is for MR job "unit" testing as in making sure that your MR job
> > is ok and/or to allow you to debug it locally before scale deployment.
> >
> > So, long story short - they are very different ;) Herriot can do
> > intricate fault injection and can work closely with a deployed cluster
> > (say control Hadoop nodes and daemons); MRUnit is focused on MR jobs
> > testing.
> >
> > Hope it helps.
> > --
> > Take care,
> > Konstantin (Cos) Boudnik
> >
> >
> > On Wed, Feb 2, 2011 at 05:44, Edson Ramiro <er...@gmail.com> wrote:
> > > Hi all,
> > >
> > > Plz, could you explain me the difference between MRUnit and Herriot?
> > >
> > > I've read the documentation of both and they seem very similar to me.
> > >
> > > Is Herriot an evolution of MRUnit?
> > >
> > > What can Herriot do that MRUnit can't?
> > >
> > > Thanks in Advance
> > >
> > > --
> > > Edson Ramiro Lucas Filho
> > > {skype, twitter, gtalk}: erlfilho
> > > http://www.inf.ufpr.br/erlf07/
> > >
> >
Re: MRUnit and Herriot
Posted by Edson Ramiro <er...@gmail.com>.
Thank you a lot Konstantin, you cleared my mind.
So, Herriot is a framework designed to test Hadoop as a whole, and (IMHO) is
a tool for help Hadoop developers and not for who is developing MR programs,
but can we use Herriot to do unit, integration or other tests on our MR
jobs?
Do you know another test tool or test framework for Hadoop?
Thanks in Advance
--
Edson Ramiro Lucas Filho
{skype, twitter, gtalk}: erlfilho
http://www.inf.ufpr.br/erlf07/
On Wed, Feb 2, 2011 at 4:58 PM, Konstantin Boudnik <co...@apache.org> wrote:
> (Moving to common-user where this belongs)
>
> Herriot is system test framework which runs against a real physical
> cluster deployed with a specially crafted build of Hadoop. That
> instrumented build of provides an extra APIs not available in Hadoop
> otherwise. These APIs are created to facilitate cluster software
> testability. Herriot isn't limited by MR but also covered (although in
> a somewhat lesser extend) HDFS side of Hadoop.
>
> MRunit is for MR job "unit" testing as in making sure that your MR job
> is ok and/or to allow you to debug it locally before scale deployment.
>
> So, long story short - they are very different ;) Herriot can do
> intricate fault injection and can work closely with a deployed cluster
> (say control Hadoop nodes and daemons); MRUnit is focused on MR jobs
> testing.
>
> Hope it helps.
> --
> Take care,
> Konstantin (Cos) Boudnik
>
>
> On Wed, Feb 2, 2011 at 05:44, Edson Ramiro <er...@gmail.com> wrote:
> > Hi all,
> >
> > Plz, could you explain me the difference between MRUnit and Herriot?
> >
> > I've read the documentation of both and they seem very similar to me.
> >
> > Is Herriot an evolution of MRUnit?
> >
> > What can Herriot do that MRUnit can't?
> >
> > Thanks in Advance
> >
> > --
> > Edson Ramiro Lucas Filho
> > {skype, twitter, gtalk}: erlfilho
> > http://www.inf.ufpr.br/erlf07/
> >
>
Re: MRUnit and Herriot
Posted by Konstantin Boudnik <co...@apache.org>.
(Moving to common-user where this belongs)
Herriot is system test framework which runs against a real physical
cluster deployed with a specially crafted build of Hadoop. That
instrumented build of provides an extra APIs not available in Hadoop
otherwise. These APIs are created to facilitate cluster software
testability. Herriot isn't limited by MR but also covered (although in
a somewhat lesser extend) HDFS side of Hadoop.
MRunit is for MR job "unit" testing as in making sure that your MR job
is ok and/or to allow you to debug it locally before scale deployment.
So, long story short - they are very different ;) Herriot can do
intricate fault injection and can work closely with a deployed cluster
(say control Hadoop nodes and daemons); MRUnit is focused on MR jobs
testing.
Hope it helps.
--
Take care,
Konstantin (Cos) Boudnik
On Wed, Feb 2, 2011 at 05:44, Edson Ramiro <er...@gmail.com> wrote:
> Hi all,
>
> Plz, could you explain me the difference between MRUnit and Herriot?
>
> I've read the documentation of both and they seem very similar to me.
>
> Is Herriot an evolution of MRUnit?
>
> What can Herriot do that MRUnit can't?
>
> Thanks in Advance
>
> --
> Edson Ramiro Lucas Filho
> {skype, twitter, gtalk}: erlfilho
> http://www.inf.ufpr.br/erlf07/
>
Re: MRUnit and Herriot
Posted by Owen O'Malley <om...@apache.org>.
Please keep user questions off of general and use the user lists instead.
This is defined here <http://hadoop.apache.org/mailing_lists.html>.
MRUnit is for testing user's MapReduce applications. Herriot is for testing
the framework in the presence of failures.
-- Owen
On Wed, Feb 2, 2011 at 5:44 AM, Edson Ramiro <er...@gmail.com> wrote:
> Hi all,
>
> Plz, could you explain me the difference between MRUnit and Herriot?
>
> I've read the documentation of both and they seem very similar to me.
>
> Is Herriot an evolution of MRUnit?
>
> What can Herriot do that MRUnit can't?
>
> Thanks in Advance
>
> --
> Edson Ramiro Lucas Filho
> {skype, twitter, gtalk}: erlfilho
> http://www.inf.ufpr.br/erlf07/
>
Re: MRUnit and Herriot
Posted by Owen O'Malley <om...@apache.org>.
Please keep user questions off of general and use the user lists instead.
This is defined here <http://hadoop.apache.org/mailing_lists.html>.
MRUnit is for testing user's MapReduce applications. Herriot is for testing
the framework in the presence of failures.
-- Owen
On Wed, Feb 2, 2011 at 5:44 AM, Edson Ramiro <er...@gmail.com> wrote:
> Hi all,
>
> Plz, could you explain me the difference between MRUnit and Herriot?
>
> I've read the documentation of both and they seem very similar to me.
>
> Is Herriot an evolution of MRUnit?
>
> What can Herriot do that MRUnit can't?
>
> Thanks in Advance
>
> --
> Edson Ramiro Lucas Filho
> {skype, twitter, gtalk}: erlfilho
> http://www.inf.ufpr.br/erlf07/
>