You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by ch huang <ju...@gmail.com> on 2013/09/02 08:56:25 UTC
how to benchmark the whole hadoop cluster performance?
hi ,all:
i want to evaluate my hadoop cluster performance ,what tool can i
use? (TestDFSIO,nnbench?)
Re: how to benchmark the whole hadoop cluster performance?
Posted by Roman Shaposhnik <rv...@apache.org>.
There's also this: https://github.com/palominolabs/benchpress
Which I really would like to integrate into Bigtop someday
and make available as a tool for Hadoop ecosystem
benchmarking.
Thanks,
Roman.
On Mon, Sep 2, 2013 at 12:15 PM, Stephen Boesch <ja...@gmail.com> wrote:
>
> You are on the right track. TestDFSIO and TeraGen/Sort provide good
> characterization of IO and shuffle /sort performance. You would likely
> want to run/save dstat (/vmstat/iostat/..) info on the individual nodes as
> well.
>
> HiBench does provide additional useful characterizations such as mixed
> workloads using typical hadoop ecosystem tools.
>
>
> 2013/9/2 Ravi Kiran <ra...@gmail.com>
>>
>> You can also look at
>> a ) https://github.com/intel-hadoop/HiBench
>>
>>
>> Regards
>> Ravi Magham
>>
>>
>> On Mon, Sep 2, 2013 at 12:26 PM, ch huang <ju...@gmail.com> wrote:
>>>
>>> hi ,all:
>>> i want to evaluate my hadoop cluster performance ,what tool can i
>>> use? (TestDFSIO,nnbench?)
>>
>>
>
Re: how to benchmark the whole hadoop cluster performance?
Posted by Roman Shaposhnik <rv...@apache.org>.
There's also this: https://github.com/palominolabs/benchpress
Which I really would like to integrate into Bigtop someday
and make available as a tool for Hadoop ecosystem
benchmarking.
Thanks,
Roman.
On Mon, Sep 2, 2013 at 12:15 PM, Stephen Boesch <ja...@gmail.com> wrote:
>
> You are on the right track. TestDFSIO and TeraGen/Sort provide good
> characterization of IO and shuffle /sort performance. You would likely
> want to run/save dstat (/vmstat/iostat/..) info on the individual nodes as
> well.
>
> HiBench does provide additional useful characterizations such as mixed
> workloads using typical hadoop ecosystem tools.
>
>
> 2013/9/2 Ravi Kiran <ra...@gmail.com>
>>
>> You can also look at
>> a ) https://github.com/intel-hadoop/HiBench
>>
>>
>> Regards
>> Ravi Magham
>>
>>
>> On Mon, Sep 2, 2013 at 12:26 PM, ch huang <ju...@gmail.com> wrote:
>>>
>>> hi ,all:
>>> i want to evaluate my hadoop cluster performance ,what tool can i
>>> use? (TestDFSIO,nnbench?)
>>
>>
>
Re: how to benchmark the whole hadoop cluster performance?
Posted by Roman Shaposhnik <rv...@apache.org>.
There's also this: https://github.com/palominolabs/benchpress
Which I really would like to integrate into Bigtop someday
and make available as a tool for Hadoop ecosystem
benchmarking.
Thanks,
Roman.
On Mon, Sep 2, 2013 at 12:15 PM, Stephen Boesch <ja...@gmail.com> wrote:
>
> You are on the right track. TestDFSIO and TeraGen/Sort provide good
> characterization of IO and shuffle /sort performance. You would likely
> want to run/save dstat (/vmstat/iostat/..) info on the individual nodes as
> well.
>
> HiBench does provide additional useful characterizations such as mixed
> workloads using typical hadoop ecosystem tools.
>
>
> 2013/9/2 Ravi Kiran <ra...@gmail.com>
>>
>> You can also look at
>> a ) https://github.com/intel-hadoop/HiBench
>>
>>
>> Regards
>> Ravi Magham
>>
>>
>> On Mon, Sep 2, 2013 at 12:26 PM, ch huang <ju...@gmail.com> wrote:
>>>
>>> hi ,all:
>>> i want to evaluate my hadoop cluster performance ,what tool can i
>>> use? (TestDFSIO,nnbench?)
>>
>>
>
Re: how to benchmark the whole hadoop cluster performance?
Posted by Roman Shaposhnik <rv...@apache.org>.
There's also this: https://github.com/palominolabs/benchpress
Which I really would like to integrate into Bigtop someday
and make available as a tool for Hadoop ecosystem
benchmarking.
Thanks,
Roman.
On Mon, Sep 2, 2013 at 12:15 PM, Stephen Boesch <ja...@gmail.com> wrote:
>
> You are on the right track. TestDFSIO and TeraGen/Sort provide good
> characterization of IO and shuffle /sort performance. You would likely
> want to run/save dstat (/vmstat/iostat/..) info on the individual nodes as
> well.
>
> HiBench does provide additional useful characterizations such as mixed
> workloads using typical hadoop ecosystem tools.
>
>
> 2013/9/2 Ravi Kiran <ra...@gmail.com>
>>
>> You can also look at
>> a ) https://github.com/intel-hadoop/HiBench
>>
>>
>> Regards
>> Ravi Magham
>>
>>
>> On Mon, Sep 2, 2013 at 12:26 PM, ch huang <ju...@gmail.com> wrote:
>>>
>>> hi ,all:
>>> i want to evaluate my hadoop cluster performance ,what tool can i
>>> use? (TestDFSIO,nnbench?)
>>
>>
>
Re: how to benchmark the whole hadoop cluster performance?
Posted by Stephen Boesch <ja...@gmail.com>.
You are on the right track. TestDFSIO and TeraGen/Sort provide good
characterization of IO and shuffle /sort performance. You would likely
want to run/save dstat (/vmstat/iostat/..) info on the individual nodes as
well.
HiBench does provide additional useful characterizations such as mixed
workloads using typical hadoop ecosystem tools.
2013/9/2 Ravi Kiran <ra...@gmail.com>
> You can also look at
> a ) https://github.com/intel-hadoop/HiBench
>
>
> Regards
> Ravi Magham
>
>
> On Mon, Sep 2, 2013 at 12:26 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi ,all:
>> i want to evaluate my hadoop cluster performance ,what tool can i
>> use? (TestDFSIO,nnbench?)
>>
>
>
Re: how to benchmark the whole hadoop cluster performance?
Posted by Stephen Boesch <ja...@gmail.com>.
You are on the right track. TestDFSIO and TeraGen/Sort provide good
characterization of IO and shuffle /sort performance. You would likely
want to run/save dstat (/vmstat/iostat/..) info on the individual nodes as
well.
HiBench does provide additional useful characterizations such as mixed
workloads using typical hadoop ecosystem tools.
2013/9/2 Ravi Kiran <ra...@gmail.com>
> You can also look at
> a ) https://github.com/intel-hadoop/HiBench
>
>
> Regards
> Ravi Magham
>
>
> On Mon, Sep 2, 2013 at 12:26 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi ,all:
>> i want to evaluate my hadoop cluster performance ,what tool can i
>> use? (TestDFSIO,nnbench?)
>>
>
>
Re: how to benchmark the whole hadoop cluster performance?
Posted by Stephen Boesch <ja...@gmail.com>.
You are on the right track. TestDFSIO and TeraGen/Sort provide good
characterization of IO and shuffle /sort performance. You would likely
want to run/save dstat (/vmstat/iostat/..) info on the individual nodes as
well.
HiBench does provide additional useful characterizations such as mixed
workloads using typical hadoop ecosystem tools.
2013/9/2 Ravi Kiran <ra...@gmail.com>
> You can also look at
> a ) https://github.com/intel-hadoop/HiBench
>
>
> Regards
> Ravi Magham
>
>
> On Mon, Sep 2, 2013 at 12:26 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi ,all:
>> i want to evaluate my hadoop cluster performance ,what tool can i
>> use? (TestDFSIO,nnbench?)
>>
>
>
Re: how to benchmark the whole hadoop cluster performance?
Posted by Stephen Boesch <ja...@gmail.com>.
You are on the right track. TestDFSIO and TeraGen/Sort provide good
characterization of IO and shuffle /sort performance. You would likely
want to run/save dstat (/vmstat/iostat/..) info on the individual nodes as
well.
HiBench does provide additional useful characterizations such as mixed
workloads using typical hadoop ecosystem tools.
2013/9/2 Ravi Kiran <ra...@gmail.com>
> You can also look at
> a ) https://github.com/intel-hadoop/HiBench
>
>
> Regards
> Ravi Magham
>
>
> On Mon, Sep 2, 2013 at 12:26 PM, ch huang <ju...@gmail.com> wrote:
>
>> hi ,all:
>> i want to evaluate my hadoop cluster performance ,what tool can i
>> use? (TestDFSIO,nnbench?)
>>
>
>
Re: how to benchmark the whole hadoop cluster performance?
Posted by Ravi Kiran <ra...@gmail.com>.
You can also look at
a ) https://github.com/intel-hadoop/HiBench
Regards
Ravi Magham
On Mon, Sep 2, 2013 at 12:26 PM, ch huang <ju...@gmail.com> wrote:
> hi ,all:
> i want to evaluate my hadoop cluster performance ,what tool can i
> use? (TestDFSIO,nnbench?)
>
Re: how to benchmark the whole hadoop cluster performance?
Posted by Ravi Kiran <ra...@gmail.com>.
You can also look at
a ) https://github.com/intel-hadoop/HiBench
Regards
Ravi Magham
On Mon, Sep 2, 2013 at 12:26 PM, ch huang <ju...@gmail.com> wrote:
> hi ,all:
> i want to evaluate my hadoop cluster performance ,what tool can i
> use? (TestDFSIO,nnbench?)
>
Re: how to benchmark the whole hadoop cluster performance?
Posted by Ravi Kiran <ra...@gmail.com>.
You can also look at
a ) https://github.com/intel-hadoop/HiBench
Regards
Ravi Magham
On Mon, Sep 2, 2013 at 12:26 PM, ch huang <ju...@gmail.com> wrote:
> hi ,all:
> i want to evaluate my hadoop cluster performance ,what tool can i
> use? (TestDFSIO,nnbench?)
>
Re: how to benchmark the whole hadoop cluster performance?
Posted by Ravi Kiran <ra...@gmail.com>.
You can also look at
a ) https://github.com/intel-hadoop/HiBench
Regards
Ravi Magham
On Mon, Sep 2, 2013 at 12:26 PM, ch huang <ju...@gmail.com> wrote:
> hi ,all:
> i want to evaluate my hadoop cluster performance ,what tool can i
> use? (TestDFSIO,nnbench?)
>