You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@bigtop.apache.org by Jay Vyas <ja...@gmail.com> on 2013/08/28 01:26:25 UTC
Terasort/Teragen in smokes
Hi guys:
I run TeraSort/TeraGen as additions to bigtop in some shell scripts.
Any interest in these as an update to TestHadoopExamples in the MapReduce
smokes?
If so I could patch them in :)
let me know and I'll put in a JIRA !
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Terasort/Teragen in smokes
Posted by Konstantin Boudnik <co...@apache.org>.
Please do, dude!
On Tue, Aug 27, 2013 at 07:26PM, Jay Vyas wrote:
> Hi guys:
>
> I run TeraSort/TeraGen as additions to bigtop in some shell scripts.
>
> Any interest in these as an update to TestHadoopExamples in the MapReduce
> smokes?
>
> If so I could patch them in :)
>
> let me know and I'll put in a JIRA !
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
Re: Terasort/Teragen in smokes
Posted by Konstantin Boudnik <co...@apache.org>.
Thanks Jay, I will try to review it today!
On Thu, Sep 05, 2013 at 01:45PM, Jay Vyas wrote:
> Hi bigtop!
>
> The 1057 patch is now up to date with teragen/terasort benchmarks.
> See https://issues.apache.org/jira/browse/BIGTOP-1057 . Let me know if
> this looks okay.
>
> Later we can open another ticket for the YCSB benchmarks which will also be
> a great, and simple smoke to add for smoke testing HBase.
>
> On Wed, Aug 28, 2013 at 12:02 AM, Konstantin Boudnik <co...@apache.org> wrote:
>
> > On Tue, Aug 27, 2013 at 10:18PM, Jay Vyas wrote:
> > > hmm ok. now. Thinking about teragen makes me think of benchmarking..
> > >
> > > In the longer term we could add benchmarking jobs to all the submodules
> > not
> > > just mapreduce. For example there are hi bench and ycsb workloads which
> > > might be usable or pulled in as bigtop components ... Iff of course
> > > benchmarking is in the cards for bigtop?
> >
> > It indeed is!
> >
> > I think doing tera-gen/sort so it can be parameterized will provide a good
> > basis for future benchmarking (as a bit of reflection: I have did
> > simplistic
> > yet efficient way of benchmarking HDFS and MR a couple years ago, but my
> > employer back then has never let it go into the open. Go figure...)
> >
> > And I have a way of building YCSB against a particular version of Hadoop,
> > so I
> > guess I will have it packaged as a benchmarking test pretty soon.
> >
> > Cos
> >
> > > On Aug 27, 2013, at 7:39 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> > >
> > > > On Tue, Aug 27, 2013 at 4:26 PM, Jay Vyas <ja...@gmail.com>
> > wrote:
> > > >> Hi guys:
> > > >>
> > > >> I run TeraSort/TeraGen as additions to bigtop in some shell scripts.
> > > >>
> > > >> Any interest in these as an update to TestHadoopExamples in the
> > MapReduce
> > > >> smokes?
> > > >>
> > > >> If so I could patch them in :)
> > > >
> > > > Sure! Sounds like a useful addition.
> > > >
> > > > Thanks,
> > > > Roman.
> >
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
Re: Terasort/Teragen in smokes
Posted by Konstantin Boudnik <co...@apache.org>.
Thanks Jay, I will try to review it today!
On Thu, Sep 05, 2013 at 01:45PM, Jay Vyas wrote:
> Hi bigtop!
>
> The 1057 patch is now up to date with teragen/terasort benchmarks.
> See https://issues.apache.org/jira/browse/BIGTOP-1057 . Let me know if
> this looks okay.
>
> Later we can open another ticket for the YCSB benchmarks which will also be
> a great, and simple smoke to add for smoke testing HBase.
>
> On Wed, Aug 28, 2013 at 12:02 AM, Konstantin Boudnik <co...@apache.org> wrote:
>
> > On Tue, Aug 27, 2013 at 10:18PM, Jay Vyas wrote:
> > > hmm ok. now. Thinking about teragen makes me think of benchmarking..
> > >
> > > In the longer term we could add benchmarking jobs to all the submodules
> > not
> > > just mapreduce. For example there are hi bench and ycsb workloads which
> > > might be usable or pulled in as bigtop components ... Iff of course
> > > benchmarking is in the cards for bigtop?
> >
> > It indeed is!
> >
> > I think doing tera-gen/sort so it can be parameterized will provide a good
> > basis for future benchmarking (as a bit of reflection: I have did
> > simplistic
> > yet efficient way of benchmarking HDFS and MR a couple years ago, but my
> > employer back then has never let it go into the open. Go figure...)
> >
> > And I have a way of building YCSB against a particular version of Hadoop,
> > so I
> > guess I will have it packaged as a benchmarking test pretty soon.
> >
> > Cos
> >
> > > On Aug 27, 2013, at 7:39 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> > >
> > > > On Tue, Aug 27, 2013 at 4:26 PM, Jay Vyas <ja...@gmail.com>
> > wrote:
> > > >> Hi guys:
> > > >>
> > > >> I run TeraSort/TeraGen as additions to bigtop in some shell scripts.
> > > >>
> > > >> Any interest in these as an update to TestHadoopExamples in the
> > MapReduce
> > > >> smokes?
> > > >>
> > > >> If so I could patch them in :)
> > > >
> > > > Sure! Sounds like a useful addition.
> > > >
> > > > Thanks,
> > > > Roman.
> >
>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
Re: Terasort/Teragen in smokes
Posted by Jay Vyas <ja...@gmail.com>.
Hi bigtop!
The 1057 patch is now up to date with teragen/terasort benchmarks.
See https://issues.apache.org/jira/browse/BIGTOP-1057 . Let me know if
this looks okay.
Later we can open another ticket for the YCSB benchmarks which will also be
a great, and simple smoke to add for smoke testing HBase.
On Wed, Aug 28, 2013 at 12:02 AM, Konstantin Boudnik <co...@apache.org> wrote:
> On Tue, Aug 27, 2013 at 10:18PM, Jay Vyas wrote:
> > hmm ok. now. Thinking about teragen makes me think of benchmarking..
> >
> > In the longer term we could add benchmarking jobs to all the submodules
> not
> > just mapreduce. For example there are hi bench and ycsb workloads which
> > might be usable or pulled in as bigtop components ... Iff of course
> > benchmarking is in the cards for bigtop?
>
> It indeed is!
>
> I think doing tera-gen/sort so it can be parameterized will provide a good
> basis for future benchmarking (as a bit of reflection: I have did
> simplistic
> yet efficient way of benchmarking HDFS and MR a couple years ago, but my
> employer back then has never let it go into the open. Go figure...)
>
> And I have a way of building YCSB against a particular version of Hadoop,
> so I
> guess I will have it packaged as a benchmarking test pretty soon.
>
> Cos
>
> > On Aug 27, 2013, at 7:39 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> >
> > > On Tue, Aug 27, 2013 at 4:26 PM, Jay Vyas <ja...@gmail.com>
> wrote:
> > >> Hi guys:
> > >>
> > >> I run TeraSort/TeraGen as additions to bigtop in some shell scripts.
> > >>
> > >> Any interest in these as an update to TestHadoopExamples in the
> MapReduce
> > >> smokes?
> > >>
> > >> If so I could patch them in :)
> > >
> > > Sure! Sounds like a useful addition.
> > >
> > > Thanks,
> > > Roman.
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Terasort/Teragen in smokes
Posted by Jay Vyas <ja...@gmail.com>.
Hi bigtop!
The 1057 patch is now up to date with teragen/terasort benchmarks.
See https://issues.apache.org/jira/browse/BIGTOP-1057 . Let me know if
this looks okay.
Later we can open another ticket for the YCSB benchmarks which will also be
a great, and simple smoke to add for smoke testing HBase.
On Wed, Aug 28, 2013 at 12:02 AM, Konstantin Boudnik <co...@apache.org> wrote:
> On Tue, Aug 27, 2013 at 10:18PM, Jay Vyas wrote:
> > hmm ok. now. Thinking about teragen makes me think of benchmarking..
> >
> > In the longer term we could add benchmarking jobs to all the submodules
> not
> > just mapreduce. For example there are hi bench and ycsb workloads which
> > might be usable or pulled in as bigtop components ... Iff of course
> > benchmarking is in the cards for bigtop?
>
> It indeed is!
>
> I think doing tera-gen/sort so it can be parameterized will provide a good
> basis for future benchmarking (as a bit of reflection: I have did
> simplistic
> yet efficient way of benchmarking HDFS and MR a couple years ago, but my
> employer back then has never let it go into the open. Go figure...)
>
> And I have a way of building YCSB against a particular version of Hadoop,
> so I
> guess I will have it packaged as a benchmarking test pretty soon.
>
> Cos
>
> > On Aug 27, 2013, at 7:39 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> >
> > > On Tue, Aug 27, 2013 at 4:26 PM, Jay Vyas <ja...@gmail.com>
> wrote:
> > >> Hi guys:
> > >>
> > >> I run TeraSort/TeraGen as additions to bigtop in some shell scripts.
> > >>
> > >> Any interest in these as an update to TestHadoopExamples in the
> MapReduce
> > >> smokes?
> > >>
> > >> If so I could patch them in :)
> > >
> > > Sure! Sounds like a useful addition.
> > >
> > > Thanks,
> > > Roman.
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Terasort/Teragen in smokes
Posted by Konstantin Boudnik <co...@apache.org>.
On Tue, Aug 27, 2013 at 10:18PM, Jay Vyas wrote:
> hmm ok. now. Thinking about teragen makes me think of benchmarking..
>
> In the longer term we could add benchmarking jobs to all the submodules not
> just mapreduce. For example there are hi bench and ycsb workloads which
> might be usable or pulled in as bigtop components ... Iff of course
> benchmarking is in the cards for bigtop?
It indeed is!
I think doing tera-gen/sort so it can be parameterized will provide a good
basis for future benchmarking (as a bit of reflection: I have did simplistic
yet efficient way of benchmarking HDFS and MR a couple years ago, but my
employer back then has never let it go into the open. Go figure...)
And I have a way of building YCSB against a particular version of Hadoop, so I
guess I will have it packaged as a benchmarking test pretty soon.
Cos
> On Aug 27, 2013, at 7:39 PM, Roman Shaposhnik <rv...@apache.org> wrote:
>
> > On Tue, Aug 27, 2013 at 4:26 PM, Jay Vyas <ja...@gmail.com> wrote:
> >> Hi guys:
> >>
> >> I run TeraSort/TeraGen as additions to bigtop in some shell scripts.
> >>
> >> Any interest in these as an update to TestHadoopExamples in the MapReduce
> >> smokes?
> >>
> >> If so I could patch them in :)
> >
> > Sure! Sounds like a useful addition.
> >
> > Thanks,
> > Roman.
Re: Terasort/Teragen in smokes
Posted by Konstantin Boudnik <co...@apache.org>.
On Tue, Aug 27, 2013 at 10:18PM, Jay Vyas wrote:
> hmm ok. now. Thinking about teragen makes me think of benchmarking..
>
> In the longer term we could add benchmarking jobs to all the submodules not
> just mapreduce. For example there are hi bench and ycsb workloads which
> might be usable or pulled in as bigtop components ... Iff of course
> benchmarking is in the cards for bigtop?
It indeed is!
I think doing tera-gen/sort so it can be parameterized will provide a good
basis for future benchmarking (as a bit of reflection: I have did simplistic
yet efficient way of benchmarking HDFS and MR a couple years ago, but my
employer back then has never let it go into the open. Go figure...)
And I have a way of building YCSB against a particular version of Hadoop, so I
guess I will have it packaged as a benchmarking test pretty soon.
Cos
> On Aug 27, 2013, at 7:39 PM, Roman Shaposhnik <rv...@apache.org> wrote:
>
> > On Tue, Aug 27, 2013 at 4:26 PM, Jay Vyas <ja...@gmail.com> wrote:
> >> Hi guys:
> >>
> >> I run TeraSort/TeraGen as additions to bigtop in some shell scripts.
> >>
> >> Any interest in these as an update to TestHadoopExamples in the MapReduce
> >> smokes?
> >>
> >> If so I could patch them in :)
> >
> > Sure! Sounds like a useful addition.
> >
> > Thanks,
> > Roman.
Re: Terasort/Teragen in smokes
Posted by Jay Vyas <ja...@gmail.com>.
hmm ok. now. Thinking about teragen makes me think of benchmarking..
In the longer term we could add benchmarking jobs to all the submodules not just mapreduce. For example there are hi bench and ycsb workloads which might be usable or pulled in as bigtop components ... Iff of course benchmarking is in the cards for bigtop?
On Aug 27, 2013, at 7:39 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> On Tue, Aug 27, 2013 at 4:26 PM, Jay Vyas <ja...@gmail.com> wrote:
>> Hi guys:
>>
>> I run TeraSort/TeraGen as additions to bigtop in some shell scripts.
>>
>> Any interest in these as an update to TestHadoopExamples in the MapReduce
>> smokes?
>>
>> If so I could patch them in :)
>
> Sure! Sounds like a useful addition.
>
> Thanks,
> Roman.
Re: Terasort/Teragen in smokes
Posted by Roman Shaposhnik <rv...@apache.org>.
On Tue, Aug 27, 2013 at 4:26 PM, Jay Vyas <ja...@gmail.com> wrote:
> Hi guys:
>
> I run TeraSort/TeraGen as additions to bigtop in some shell scripts.
>
> Any interest in these as an update to TestHadoopExamples in the MapReduce
> smokes?
>
> If so I could patch them in :)
Sure! Sounds like a useful addition.
Thanks,
Roman.