Re: Terasort/Teragen in smokes

[Jay Vyas <ja...@gmail.com>]

Hi bigtop!

The 1057 patch is now up to date with teragen/terasort benchmarks.
See https://issues.apache.org/jira/browse/BIGTOP-1057 .   Let me know if
this looks okay.

Later we can open another ticket for the YCSB benchmarks which will also be
a great, and simple smoke to add for smoke testing HBase.

On Wed, Aug 28, 2013 at 12:02 AM, Konstantin Boudnik <co...@apache.org> wrote:

> On Tue, Aug 27, 2013 at 10:18PM, Jay Vyas wrote:
> > hmm ok. now. Thinking about teragen makes me think of benchmarking..
> >
> >  In the longer term we could add benchmarking jobs to all the submodules
> not
> >  just mapreduce.  For example there are hi bench and ycsb workloads which
> >  might be usable or pulled in as bigtop components ... Iff of course
> >  benchmarking is in the cards for bigtop?
>
> It indeed is!
>
> I think doing tera-gen/sort so it can be parameterized will provide a good
> basis for future benchmarking (as a bit of reflection: I have did
> simplistic
> yet efficient way of benchmarking HDFS and MR a couple years ago, but my
> employer back then has never let it go into the open. Go figure...)
>
> And I have a way of building YCSB against a particular version of Hadoop,
> so I
> guess I will have it packaged as a benchmarking test pretty soon.
>
> Cos
>
> > On Aug 27, 2013, at 7:39 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> >
> > > On Tue, Aug 27, 2013 at 4:26 PM, Jay Vyas <ja...@gmail.com>
> wrote:
> > >> Hi guys:
> > >>
> > >> I run TeraSort/TeraGen as additions to bigtop in some shell scripts.
> > >>
> > >> Any interest in these as an update to TestHadoopExamples in the
> MapReduce
> > >> smokes?
> > >>
> > >> If so I could patch them in :)
> > >
> > > Sure! Sounds like a useful addition.
> > >
> > > Thanks,
> > > Roman.
>



-- 
Jay Vyas
http://jayunit100.blogspot.com

Re: Terasort/Teragen in smokes

[Konstantin Boudnik <co...@apache.org>]

Thanks Jay, I will try to review it today!

On Thu, Sep 05, 2013 at 01:45PM, Jay Vyas wrote:
> Hi bigtop!
> 
> The 1057 patch is now up to date with teragen/terasort benchmarks.
> See https://issues.apache.org/jira/browse/BIGTOP-1057 .   Let me know if
> this looks okay.
> 
> Later we can open another ticket for the YCSB benchmarks which will also be
> a great, and simple smoke to add for smoke testing HBase.
> 
> On Wed, Aug 28, 2013 at 12:02 AM, Konstantin Boudnik <co...@apache.org> wrote:
> 
> > On Tue, Aug 27, 2013 at 10:18PM, Jay Vyas wrote:
> > > hmm ok. now. Thinking about teragen makes me think of benchmarking..
> > >
> > >  In the longer term we could add benchmarking jobs to all the submodules
> > not
> > >  just mapreduce.  For example there are hi bench and ycsb workloads which
> > >  might be usable or pulled in as bigtop components ... Iff of course
> > >  benchmarking is in the cards for bigtop?
> >
> > It indeed is!
> >
> > I think doing tera-gen/sort so it can be parameterized will provide a good
> > basis for future benchmarking (as a bit of reflection: I have did
> > simplistic
> > yet efficient way of benchmarking HDFS and MR a couple years ago, but my
> > employer back then has never let it go into the open. Go figure...)
> >
> > And I have a way of building YCSB against a particular version of Hadoop,
> > so I
> > guess I will have it packaged as a benchmarking test pretty soon.
> >
> > Cos
> >
> > > On Aug 27, 2013, at 7:39 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> > >
> > > > On Tue, Aug 27, 2013 at 4:26 PM, Jay Vyas <ja...@gmail.com>
> > wrote:
> > > >> Hi guys:
> > > >>
> > > >> I run TeraSort/TeraGen as additions to bigtop in some shell scripts.
> > > >>
> > > >> Any interest in these as an update to TestHadoopExamples in the
> > MapReduce
> > > >> smokes?
> > > >>
> > > >> If so I could patch them in :)
> > > >
> > > > Sure! Sounds like a useful addition.
> > > >
> > > > Thanks,
> > > > Roman.
> >
> 
> 
> 
> -- 
> Jay Vyas
> http://jayunit100.blogspot.com

Re: Terasort/Teragen in smokes

[Konstantin Boudnik <co...@apache.org>]

Thanks Jay, I will try to review it today!

On Thu, Sep 05, 2013 at 01:45PM, Jay Vyas wrote:
> Hi bigtop!
> 
> The 1057 patch is now up to date with teragen/terasort benchmarks.
> See https://issues.apache.org/jira/browse/BIGTOP-1057 .   Let me know if
> this looks okay.
> 
> Later we can open another ticket for the YCSB benchmarks which will also be
> a great, and simple smoke to add for smoke testing HBase.
> 
> On Wed, Aug 28, 2013 at 12:02 AM, Konstantin Boudnik <co...@apache.org> wrote:
> 
> > On Tue, Aug 27, 2013 at 10:18PM, Jay Vyas wrote:
> > > hmm ok. now. Thinking about teragen makes me think of benchmarking..
> > >
> > >  In the longer term we could add benchmarking jobs to all the submodules
> > not
> > >  just mapreduce.  For example there are hi bench and ycsb workloads which
> > >  might be usable or pulled in as bigtop components ... Iff of course
> > >  benchmarking is in the cards for bigtop?
> >
> > It indeed is!
> >
> > I think doing tera-gen/sort so it can be parameterized will provide a good
> > basis for future benchmarking (as a bit of reflection: I have did
> > simplistic
> > yet efficient way of benchmarking HDFS and MR a couple years ago, but my
> > employer back then has never let it go into the open. Go figure...)
> >
> > And I have a way of building YCSB against a particular version of Hadoop,
> > so I
> > guess I will have it packaged as a benchmarking test pretty soon.
> >
> > Cos
> >
> > > On Aug 27, 2013, at 7:39 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> > >
> > > > On Tue, Aug 27, 2013 at 4:26 PM, Jay Vyas <ja...@gmail.com>
> > wrote:
> > > >> Hi guys:
> > > >>
> > > >> I run TeraSort/TeraGen as additions to bigtop in some shell scripts.
> > > >>
> > > >> Any interest in these as an update to TestHadoopExamples in the
> > MapReduce
> > > >> smokes?
> > > >>
> > > >> If so I could patch them in :)
> > > >
> > > > Sure! Sounds like a useful addition.
> > > >
> > > > Thanks,
> > > > Roman.
> >
> 
> 
> 
> -- 
> Jay Vyas
> http://jayunit100.blogspot.com