You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Manuel Sopena Ballesteros <ma...@garvan.org.au> on 2018/01/30 01:08:35 UTC

HDFS latency and bandwidth/speed

Hi all,

I am going to start working on HDFS, how could I test HDFS latency and speed? Is there an ioping command and or hdpram or fio I can use in HDFS?

Thank you very much

Manuel Sopena Ballesteros | Big data Engineer
Garvan Institute of Medical Research
The Kinghorn Cancer Centre, 370 Victoria Street, Darlinghurst, NSW 2010
T: + 61 (0)2 9355 5760 | F: +61 (0)2 9295 8507 | E: manuel.sb@garvan.org.au<ma...@garvan.org.au>

NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.

RE: HDFS latency and bandwidth/speed

Posted by Manuel Sopena Ballesteros <ma...@garvan.org.au>.
Thank you very much Anu,

This is very useful

Manuel

From: Anu Engineer [mailto:aengineer@hortonworks.com]
Sent: Tuesday, January 30, 2018 12:25 PM
To: Manuel Sopena Ballesteros; user@hadoop.apache.org
Subject: Re: HDFS latency and bandwidth/speed

Hi Manuel,



Depending on your use case: There are several tools. Unfortunately, most of them need some familiarity with HDFS.

Here is a quick set of links that google returns.

https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-common/Benchmarking.html

An old blog, but most of these applications work. There are a set of applications that get shipped with Hadoop. Both DFSIO and Terragen are useful benchmarks.

http://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/

If this is the first time you are using HDFS, you might want to take this as an opportunity to write a small program that reads the local files and puts them on to HDFS.

When you start working against the cluster, it is the apps that matter, and having some familiarity with how applications are written will be very useful.

Thanks
Anu

From: Manuel Sopena Ballesteros <ma...@garvan.org.au>>
Date: Monday, January 29, 2018 at 5:08 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: HDFS latency and bandwidth/speed

Hi all,

I am going to start working on HDFS, how could I test HDFS latency and speed? Is there an ioping command and or hdpram or fio I can use in HDFS?

Thank you very much

Manuel Sopena Ballesteros | Big data Engineer
Garvan Institute of Medical Research
The Kinghorn Cancer Centre, 370 Victoria Street, Darlinghurst, NSW 2010
T: + 61 (0)2 9355 5760 | F: +61 (0)2 9295 8507 | E: manuel.sb@garvan.org.au<ma...@garvan.org.au>

NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.

Re: HDFS latency and bandwidth/speed

Posted by Anu Engineer <ae...@hortonworks.com>.
Hi Manuel,



Depending on your use case: There are several tools. Unfortunately, most of them need some familiarity with HDFS.

Here is a quick set of links that google returns.

https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-common/Benchmarking.html

An old blog, but most of these applications work. There are a set of applications that get shipped with Hadoop. Both DFSIO and Terragen are useful benchmarks.

http://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/

If this is the first time you are using HDFS, you might want to take this as an opportunity to write a small program that reads the local files and puts them on to HDFS.

When you start working against the cluster, it is the apps that matter, and having some familiarity with how applications are written will be very useful.

Thanks
Anu

From: Manuel Sopena Ballesteros <ma...@garvan.org.au>
Date: Monday, January 29, 2018 at 5:08 PM
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
Subject: HDFS latency and bandwidth/speed

Hi all,

I am going to start working on HDFS, how could I test HDFS latency and speed? Is there an ioping command and or hdpram or fio I can use in HDFS?

Thank you very much

Manuel Sopena Ballesteros | Big data Engineer
Garvan Institute of Medical Research
The Kinghorn Cancer Centre, 370 Victoria Street, Darlinghurst, NSW 2010
T: + 61 (0)2 9355 5760 | F: +61 (0)2 9295 8507 | E: manuel.sb@garvan.org.au<ma...@garvan.org.au>

NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.