You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Mark Kerzner <ma...@shmsoft.com> on 2013/06/06 00:17:21 UTC
How to test the performance of NN?
Hi,
I am trying to create a more efficient namenode, and for that I need to the
standard distribution, and then compare it to my version.
Which benchmark should I run? I am doing nnbench, but it is not telling me
anything about performance, only about potential failures.
Thank you.
Sincerely,
Mark
Re: How to test the performance of NN?
Posted by Mark Kerzner <ma...@shmsoft.com>.
Awesome advice, Ivan! It works for me now.
Thank you,
Mark
On Thu, Jun 6, 2013 at 12:18 AM, Ivan Mitic <iv...@microsoft.com> wrote:
> Hi Mark,****
>
> ** **
>
> Your NNBench output does not look OK to me. Below is an example NNBench
> cmd line I used some time ago and its output.****
>
> ** **
>
> Btw, what I found useful in testing the disk IO performance for namenode
> operations (others might find this unusual) is the TestEditLog unittest. I
> would modify the test as appropriate and run the simulation. This was a
> quick micro-benchmark I used to compare a couple of devices. Note that
> you’ll have to modify the test in trunk and remove the line with
> EditLogFileOutputStream.setShouldSkipFsyncForTesting(), otherwise you’re
> not testing the real thing.****
>
> ** **
>
> ** **
>
> hadoop.cmd jar hadoop-test-1.1.0-SNAPSHOT.jar nnbench -operation
> create_write -maps 32 -reduces 1 -blockSize 1 -bytesToWrite 20
> -bytesPerChecksum 1 -numberOfFiles 500 -replicationFactorPerFile 1****
>
> ** **
>
> ** **
>
> -------------- NNBench -------------- :****
>
> Version: NameNode Benchmark 0.4****
>
> Date & time: 2012-12-18 07:29:20,832****
>
> ** **
>
> Test Operation: create_write****
>
> Start time: 2012-12-18 07:25:36,111****
>
> Maps to run: 32****
>
> Reduces to run: 1****
>
> Block Size (bytes): 1****
>
> Bytes to write: 20****
>
> Bytes per checksum: 1****
>
> Number of files: 500****
>
> Replication factor: 1****
>
> Successful file operations: 16000****
>
> ** **
>
> # maps that missed the barrier: 0****
>
> # exceptions: 0****
>
> ** **
>
> TPS: Create/Write/Close: 153****
>
> Avg exec time (ms): Create/Write/Close: 301.6148125****
>
> Avg Lat (ms): Create/Write: 68.25625****
>
> Avg Lat (ms): Close: 233.245****
>
> ** **
>
> RAW DATA: AL Total #1: 1092100****
>
> RAW DATA: AL Total #2: 3731920****
>
> RAW DATA: TPS Total (ms): 4825837****
>
> RAW DATA: Longest Map Time (ms): 208692.0****
>
> RAW DATA: Late maps: 0****
>
> RAW DATA: # of exceptions: 0****
>
> ** **
>
> Hope this helps,****
>
> Ivan****
>
> ** **
>
> *From:* Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
> *Sent:* Wednesday, June 05, 2013 5:16 PM
> *To:* Hadoop User
> *Subject:* Re: How to test the performance of NN?****
>
> ** **
>
> Hi, Ivan,****
>
> thank you for your willingness to help. First, what do I mean by a more
> efficient NN? - I am experimenting with Kove <http://kove.com/> device,
> persistent memory. I want to do one of the two things:****
>
> 1. Use it for FSDirectory, with the expectation of very fast writes, to
> see if this improves performance.****
>
> 2. Use it for BlockMaps, not for performance, but for fault-tolerance.****
>
> To prepare for this, I want to run some benchmarks on the given install
> just to get experience running it. Then I will run it on a 3-node cluster
> without Kove, and then with Kove.****
>
> Here is what I got from nnbench, and I am trying to find what I should
> watch for to measure NN performance. Does this all make sense?****
>
> Thank you,
> Mark
>
>
> 13/06/05 16:50:16 INFO hdfs.NNBench: -------------- NNBench --------------
> :
> 13/06/05 16:50:16 INFO hdfs.NNBench:
> Version: NameNode Benchmark 0.4
> 13/06/05 16:50:16 INFO hdfs.NNBench: Date &
> time: 2013-06-05 16:50:16,412
> 13/06/05 16:50:16 INFO hdfs.NNBench:
> 13/06/05 16:50:16 INFO hdfs.NNBench: Test
> Operation: open_read
> 13/06/05 16:50:16 INFO hdfs.NNBench: Start
> time: 2013-06-05 16:48:34,792
> 13/06/05 16:50:16 INFO hdfs.NNBench: Maps to
> run: 12
> 13/06/05 16:50:16 INFO hdfs.NNBench: Reduces to
> run: 6
> 13/06/05 16:50:16 INFO hdfs.NNBench: Block Size
> (bytes): 1
> 13/06/05 16:50:16 INFO hdfs.NNBench: Bytes to
> write: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: Bytes per
> checksum: 1
> 13/06/05 16:50:16 INFO hdfs.NNBench: Number of
> files: 1000
> 13/06/05 16:50:16 INFO hdfs.NNBench: Replication
> factor: 3
> 13/06/05 16:50:16 INFO hdfs.NNBench: Successful file
> operations: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench:
> 13/06/05 16:50:16 INFO hdfs.NNBench: # maps that missed the
> barrier: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: #
> exceptions: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench:
> 13/06/05 16:50:16 INFO hdfs.NNBench: TPS:
> Open/Read: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: Avg Exec time (ms):
> Open/Read: 0.0
> 13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms):
> Open: Infinity
> 13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms): Read:
> NaN
> 13/06/05 16:50:16 INFO hdfs.NNBench:
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total
> #1: 4665
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total
> #2: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: TPS Total
> (ms): 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Longest Map Time
> (ms): 0.0
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Late
> maps: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: # of
> exceptions: 0****
>
> ** **
>
> ** **
>
> On Wed, Jun 5, 2013 at 4:32 PM, Ivan Mitic <iv...@microsoft.com> wrote:**
> **
>
> Hi Mark,****
>
> ****
>
> NNBench is a namenode load test. Output of the test is the set of
> performance numbers, like transactions per second, average latency of
> operations, etc.****
>
> ****
>
> What do you mean by trying to create a more efficient namenode? What
> dimension are you trying to optimize? Depending on this, people on this
> list might be able to guide you to something more appropriate. ****
>
> ****
>
> Hope this helps,****
>
> Ivan ****
>
> ****
>
> *From:* Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
> *Sent:* Wednesday, June 05, 2013 3:17 PM
> *To:* Hadoop User
> *Subject:* How to test the performance of NN?****
>
> ****
>
> Hi,
>
> I am trying to create a more efficient namenode, and for that I need to
> the standard distribution, and then compare it to my version.
>
> Which benchmark should I run? I am doing nnbench, but it is not telling me
> anything about performance, only about potential failures.
>
> Thank you.****
>
> Sincerely,
> Mark****
>
> ** **
>
Re: How to test the performance of NN?
Posted by Mark Kerzner <ma...@shmsoft.com>.
Awesome advice, Ivan! It works for me now.
Thank you,
Mark
On Thu, Jun 6, 2013 at 12:18 AM, Ivan Mitic <iv...@microsoft.com> wrote:
> Hi Mark,****
>
> ** **
>
> Your NNBench output does not look OK to me. Below is an example NNBench
> cmd line I used some time ago and its output.****
>
> ** **
>
> Btw, what I found useful in testing the disk IO performance for namenode
> operations (others might find this unusual) is the TestEditLog unittest. I
> would modify the test as appropriate and run the simulation. This was a
> quick micro-benchmark I used to compare a couple of devices. Note that
> you’ll have to modify the test in trunk and remove the line with
> EditLogFileOutputStream.setShouldSkipFsyncForTesting(), otherwise you’re
> not testing the real thing.****
>
> ** **
>
> ** **
>
> hadoop.cmd jar hadoop-test-1.1.0-SNAPSHOT.jar nnbench -operation
> create_write -maps 32 -reduces 1 -blockSize 1 -bytesToWrite 20
> -bytesPerChecksum 1 -numberOfFiles 500 -replicationFactorPerFile 1****
>
> ** **
>
> ** **
>
> -------------- NNBench -------------- :****
>
> Version: NameNode Benchmark 0.4****
>
> Date & time: 2012-12-18 07:29:20,832****
>
> ** **
>
> Test Operation: create_write****
>
> Start time: 2012-12-18 07:25:36,111****
>
> Maps to run: 32****
>
> Reduces to run: 1****
>
> Block Size (bytes): 1****
>
> Bytes to write: 20****
>
> Bytes per checksum: 1****
>
> Number of files: 500****
>
> Replication factor: 1****
>
> Successful file operations: 16000****
>
> ** **
>
> # maps that missed the barrier: 0****
>
> # exceptions: 0****
>
> ** **
>
> TPS: Create/Write/Close: 153****
>
> Avg exec time (ms): Create/Write/Close: 301.6148125****
>
> Avg Lat (ms): Create/Write: 68.25625****
>
> Avg Lat (ms): Close: 233.245****
>
> ** **
>
> RAW DATA: AL Total #1: 1092100****
>
> RAW DATA: AL Total #2: 3731920****
>
> RAW DATA: TPS Total (ms): 4825837****
>
> RAW DATA: Longest Map Time (ms): 208692.0****
>
> RAW DATA: Late maps: 0****
>
> RAW DATA: # of exceptions: 0****
>
> ** **
>
> Hope this helps,****
>
> Ivan****
>
> ** **
>
> *From:* Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
> *Sent:* Wednesday, June 05, 2013 5:16 PM
> *To:* Hadoop User
> *Subject:* Re: How to test the performance of NN?****
>
> ** **
>
> Hi, Ivan,****
>
> thank you for your willingness to help. First, what do I mean by a more
> efficient NN? - I am experimenting with Kove <http://kove.com/> device,
> persistent memory. I want to do one of the two things:****
>
> 1. Use it for FSDirectory, with the expectation of very fast writes, to
> see if this improves performance.****
>
> 2. Use it for BlockMaps, not for performance, but for fault-tolerance.****
>
> To prepare for this, I want to run some benchmarks on the given install
> just to get experience running it. Then I will run it on a 3-node cluster
> without Kove, and then with Kove.****
>
> Here is what I got from nnbench, and I am trying to find what I should
> watch for to measure NN performance. Does this all make sense?****
>
> Thank you,
> Mark
>
>
> 13/06/05 16:50:16 INFO hdfs.NNBench: -------------- NNBench --------------
> :
> 13/06/05 16:50:16 INFO hdfs.NNBench:
> Version: NameNode Benchmark 0.4
> 13/06/05 16:50:16 INFO hdfs.NNBench: Date &
> time: 2013-06-05 16:50:16,412
> 13/06/05 16:50:16 INFO hdfs.NNBench:
> 13/06/05 16:50:16 INFO hdfs.NNBench: Test
> Operation: open_read
> 13/06/05 16:50:16 INFO hdfs.NNBench: Start
> time: 2013-06-05 16:48:34,792
> 13/06/05 16:50:16 INFO hdfs.NNBench: Maps to
> run: 12
> 13/06/05 16:50:16 INFO hdfs.NNBench: Reduces to
> run: 6
> 13/06/05 16:50:16 INFO hdfs.NNBench: Block Size
> (bytes): 1
> 13/06/05 16:50:16 INFO hdfs.NNBench: Bytes to
> write: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: Bytes per
> checksum: 1
> 13/06/05 16:50:16 INFO hdfs.NNBench: Number of
> files: 1000
> 13/06/05 16:50:16 INFO hdfs.NNBench: Replication
> factor: 3
> 13/06/05 16:50:16 INFO hdfs.NNBench: Successful file
> operations: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench:
> 13/06/05 16:50:16 INFO hdfs.NNBench: # maps that missed the
> barrier: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: #
> exceptions: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench:
> 13/06/05 16:50:16 INFO hdfs.NNBench: TPS:
> Open/Read: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: Avg Exec time (ms):
> Open/Read: 0.0
> 13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms):
> Open: Infinity
> 13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms): Read:
> NaN
> 13/06/05 16:50:16 INFO hdfs.NNBench:
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total
> #1: 4665
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total
> #2: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: TPS Total
> (ms): 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Longest Map Time
> (ms): 0.0
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Late
> maps: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: # of
> exceptions: 0****
>
> ** **
>
> ** **
>
> On Wed, Jun 5, 2013 at 4:32 PM, Ivan Mitic <iv...@microsoft.com> wrote:**
> **
>
> Hi Mark,****
>
> ****
>
> NNBench is a namenode load test. Output of the test is the set of
> performance numbers, like transactions per second, average latency of
> operations, etc.****
>
> ****
>
> What do you mean by trying to create a more efficient namenode? What
> dimension are you trying to optimize? Depending on this, people on this
> list might be able to guide you to something more appropriate. ****
>
> ****
>
> Hope this helps,****
>
> Ivan ****
>
> ****
>
> *From:* Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
> *Sent:* Wednesday, June 05, 2013 3:17 PM
> *To:* Hadoop User
> *Subject:* How to test the performance of NN?****
>
> ****
>
> Hi,
>
> I am trying to create a more efficient namenode, and for that I need to
> the standard distribution, and then compare it to my version.
>
> Which benchmark should I run? I am doing nnbench, but it is not telling me
> anything about performance, only about potential failures.
>
> Thank you.****
>
> Sincerely,
> Mark****
>
> ** **
>
Re: How to test the performance of NN?
Posted by Mark Kerzner <ma...@shmsoft.com>.
Awesome advice, Ivan! It works for me now.
Thank you,
Mark
On Thu, Jun 6, 2013 at 12:18 AM, Ivan Mitic <iv...@microsoft.com> wrote:
> Hi Mark,****
>
> ** **
>
> Your NNBench output does not look OK to me. Below is an example NNBench
> cmd line I used some time ago and its output.****
>
> ** **
>
> Btw, what I found useful in testing the disk IO performance for namenode
> operations (others might find this unusual) is the TestEditLog unittest. I
> would modify the test as appropriate and run the simulation. This was a
> quick micro-benchmark I used to compare a couple of devices. Note that
> you’ll have to modify the test in trunk and remove the line with
> EditLogFileOutputStream.setShouldSkipFsyncForTesting(), otherwise you’re
> not testing the real thing.****
>
> ** **
>
> ** **
>
> hadoop.cmd jar hadoop-test-1.1.0-SNAPSHOT.jar nnbench -operation
> create_write -maps 32 -reduces 1 -blockSize 1 -bytesToWrite 20
> -bytesPerChecksum 1 -numberOfFiles 500 -replicationFactorPerFile 1****
>
> ** **
>
> ** **
>
> -------------- NNBench -------------- :****
>
> Version: NameNode Benchmark 0.4****
>
> Date & time: 2012-12-18 07:29:20,832****
>
> ** **
>
> Test Operation: create_write****
>
> Start time: 2012-12-18 07:25:36,111****
>
> Maps to run: 32****
>
> Reduces to run: 1****
>
> Block Size (bytes): 1****
>
> Bytes to write: 20****
>
> Bytes per checksum: 1****
>
> Number of files: 500****
>
> Replication factor: 1****
>
> Successful file operations: 16000****
>
> ** **
>
> # maps that missed the barrier: 0****
>
> # exceptions: 0****
>
> ** **
>
> TPS: Create/Write/Close: 153****
>
> Avg exec time (ms): Create/Write/Close: 301.6148125****
>
> Avg Lat (ms): Create/Write: 68.25625****
>
> Avg Lat (ms): Close: 233.245****
>
> ** **
>
> RAW DATA: AL Total #1: 1092100****
>
> RAW DATA: AL Total #2: 3731920****
>
> RAW DATA: TPS Total (ms): 4825837****
>
> RAW DATA: Longest Map Time (ms): 208692.0****
>
> RAW DATA: Late maps: 0****
>
> RAW DATA: # of exceptions: 0****
>
> ** **
>
> Hope this helps,****
>
> Ivan****
>
> ** **
>
> *From:* Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
> *Sent:* Wednesday, June 05, 2013 5:16 PM
> *To:* Hadoop User
> *Subject:* Re: How to test the performance of NN?****
>
> ** **
>
> Hi, Ivan,****
>
> thank you for your willingness to help. First, what do I mean by a more
> efficient NN? - I am experimenting with Kove <http://kove.com/> device,
> persistent memory. I want to do one of the two things:****
>
> 1. Use it for FSDirectory, with the expectation of very fast writes, to
> see if this improves performance.****
>
> 2. Use it for BlockMaps, not for performance, but for fault-tolerance.****
>
> To prepare for this, I want to run some benchmarks on the given install
> just to get experience running it. Then I will run it on a 3-node cluster
> without Kove, and then with Kove.****
>
> Here is what I got from nnbench, and I am trying to find what I should
> watch for to measure NN performance. Does this all make sense?****
>
> Thank you,
> Mark
>
>
> 13/06/05 16:50:16 INFO hdfs.NNBench: -------------- NNBench --------------
> :
> 13/06/05 16:50:16 INFO hdfs.NNBench:
> Version: NameNode Benchmark 0.4
> 13/06/05 16:50:16 INFO hdfs.NNBench: Date &
> time: 2013-06-05 16:50:16,412
> 13/06/05 16:50:16 INFO hdfs.NNBench:
> 13/06/05 16:50:16 INFO hdfs.NNBench: Test
> Operation: open_read
> 13/06/05 16:50:16 INFO hdfs.NNBench: Start
> time: 2013-06-05 16:48:34,792
> 13/06/05 16:50:16 INFO hdfs.NNBench: Maps to
> run: 12
> 13/06/05 16:50:16 INFO hdfs.NNBench: Reduces to
> run: 6
> 13/06/05 16:50:16 INFO hdfs.NNBench: Block Size
> (bytes): 1
> 13/06/05 16:50:16 INFO hdfs.NNBench: Bytes to
> write: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: Bytes per
> checksum: 1
> 13/06/05 16:50:16 INFO hdfs.NNBench: Number of
> files: 1000
> 13/06/05 16:50:16 INFO hdfs.NNBench: Replication
> factor: 3
> 13/06/05 16:50:16 INFO hdfs.NNBench: Successful file
> operations: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench:
> 13/06/05 16:50:16 INFO hdfs.NNBench: # maps that missed the
> barrier: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: #
> exceptions: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench:
> 13/06/05 16:50:16 INFO hdfs.NNBench: TPS:
> Open/Read: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: Avg Exec time (ms):
> Open/Read: 0.0
> 13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms):
> Open: Infinity
> 13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms): Read:
> NaN
> 13/06/05 16:50:16 INFO hdfs.NNBench:
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total
> #1: 4665
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total
> #2: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: TPS Total
> (ms): 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Longest Map Time
> (ms): 0.0
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Late
> maps: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: # of
> exceptions: 0****
>
> ** **
>
> ** **
>
> On Wed, Jun 5, 2013 at 4:32 PM, Ivan Mitic <iv...@microsoft.com> wrote:**
> **
>
> Hi Mark,****
>
> ****
>
> NNBench is a namenode load test. Output of the test is the set of
> performance numbers, like transactions per second, average latency of
> operations, etc.****
>
> ****
>
> What do you mean by trying to create a more efficient namenode? What
> dimension are you trying to optimize? Depending on this, people on this
> list might be able to guide you to something more appropriate. ****
>
> ****
>
> Hope this helps,****
>
> Ivan ****
>
> ****
>
> *From:* Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
> *Sent:* Wednesday, June 05, 2013 3:17 PM
> *To:* Hadoop User
> *Subject:* How to test the performance of NN?****
>
> ****
>
> Hi,
>
> I am trying to create a more efficient namenode, and for that I need to
> the standard distribution, and then compare it to my version.
>
> Which benchmark should I run? I am doing nnbench, but it is not telling me
> anything about performance, only about potential failures.
>
> Thank you.****
>
> Sincerely,
> Mark****
>
> ** **
>
Re: How to test the performance of NN?
Posted by Mark Kerzner <ma...@shmsoft.com>.
Awesome advice, Ivan! It works for me now.
Thank you,
Mark
On Thu, Jun 6, 2013 at 12:18 AM, Ivan Mitic <iv...@microsoft.com> wrote:
> Hi Mark,****
>
> ** **
>
> Your NNBench output does not look OK to me. Below is an example NNBench
> cmd line I used some time ago and its output.****
>
> ** **
>
> Btw, what I found useful in testing the disk IO performance for namenode
> operations (others might find this unusual) is the TestEditLog unittest. I
> would modify the test as appropriate and run the simulation. This was a
> quick micro-benchmark I used to compare a couple of devices. Note that
> you’ll have to modify the test in trunk and remove the line with
> EditLogFileOutputStream.setShouldSkipFsyncForTesting(), otherwise you’re
> not testing the real thing.****
>
> ** **
>
> ** **
>
> hadoop.cmd jar hadoop-test-1.1.0-SNAPSHOT.jar nnbench -operation
> create_write -maps 32 -reduces 1 -blockSize 1 -bytesToWrite 20
> -bytesPerChecksum 1 -numberOfFiles 500 -replicationFactorPerFile 1****
>
> ** **
>
> ** **
>
> -------------- NNBench -------------- :****
>
> Version: NameNode Benchmark 0.4****
>
> Date & time: 2012-12-18 07:29:20,832****
>
> ** **
>
> Test Operation: create_write****
>
> Start time: 2012-12-18 07:25:36,111****
>
> Maps to run: 32****
>
> Reduces to run: 1****
>
> Block Size (bytes): 1****
>
> Bytes to write: 20****
>
> Bytes per checksum: 1****
>
> Number of files: 500****
>
> Replication factor: 1****
>
> Successful file operations: 16000****
>
> ** **
>
> # maps that missed the barrier: 0****
>
> # exceptions: 0****
>
> ** **
>
> TPS: Create/Write/Close: 153****
>
> Avg exec time (ms): Create/Write/Close: 301.6148125****
>
> Avg Lat (ms): Create/Write: 68.25625****
>
> Avg Lat (ms): Close: 233.245****
>
> ** **
>
> RAW DATA: AL Total #1: 1092100****
>
> RAW DATA: AL Total #2: 3731920****
>
> RAW DATA: TPS Total (ms): 4825837****
>
> RAW DATA: Longest Map Time (ms): 208692.0****
>
> RAW DATA: Late maps: 0****
>
> RAW DATA: # of exceptions: 0****
>
> ** **
>
> Hope this helps,****
>
> Ivan****
>
> ** **
>
> *From:* Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
> *Sent:* Wednesday, June 05, 2013 5:16 PM
> *To:* Hadoop User
> *Subject:* Re: How to test the performance of NN?****
>
> ** **
>
> Hi, Ivan,****
>
> thank you for your willingness to help. First, what do I mean by a more
> efficient NN? - I am experimenting with Kove <http://kove.com/> device,
> persistent memory. I want to do one of the two things:****
>
> 1. Use it for FSDirectory, with the expectation of very fast writes, to
> see if this improves performance.****
>
> 2. Use it for BlockMaps, not for performance, but for fault-tolerance.****
>
> To prepare for this, I want to run some benchmarks on the given install
> just to get experience running it. Then I will run it on a 3-node cluster
> without Kove, and then with Kove.****
>
> Here is what I got from nnbench, and I am trying to find what I should
> watch for to measure NN performance. Does this all make sense?****
>
> Thank you,
> Mark
>
>
> 13/06/05 16:50:16 INFO hdfs.NNBench: -------------- NNBench --------------
> :
> 13/06/05 16:50:16 INFO hdfs.NNBench:
> Version: NameNode Benchmark 0.4
> 13/06/05 16:50:16 INFO hdfs.NNBench: Date &
> time: 2013-06-05 16:50:16,412
> 13/06/05 16:50:16 INFO hdfs.NNBench:
> 13/06/05 16:50:16 INFO hdfs.NNBench: Test
> Operation: open_read
> 13/06/05 16:50:16 INFO hdfs.NNBench: Start
> time: 2013-06-05 16:48:34,792
> 13/06/05 16:50:16 INFO hdfs.NNBench: Maps to
> run: 12
> 13/06/05 16:50:16 INFO hdfs.NNBench: Reduces to
> run: 6
> 13/06/05 16:50:16 INFO hdfs.NNBench: Block Size
> (bytes): 1
> 13/06/05 16:50:16 INFO hdfs.NNBench: Bytes to
> write: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: Bytes per
> checksum: 1
> 13/06/05 16:50:16 INFO hdfs.NNBench: Number of
> files: 1000
> 13/06/05 16:50:16 INFO hdfs.NNBench: Replication
> factor: 3
> 13/06/05 16:50:16 INFO hdfs.NNBench: Successful file
> operations: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench:
> 13/06/05 16:50:16 INFO hdfs.NNBench: # maps that missed the
> barrier: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: #
> exceptions: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench:
> 13/06/05 16:50:16 INFO hdfs.NNBench: TPS:
> Open/Read: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: Avg Exec time (ms):
> Open/Read: 0.0
> 13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms):
> Open: Infinity
> 13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms): Read:
> NaN
> 13/06/05 16:50:16 INFO hdfs.NNBench:
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total
> #1: 4665
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total
> #2: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: TPS Total
> (ms): 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Longest Map Time
> (ms): 0.0
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Late
> maps: 0
> 13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: # of
> exceptions: 0****
>
> ** **
>
> ** **
>
> On Wed, Jun 5, 2013 at 4:32 PM, Ivan Mitic <iv...@microsoft.com> wrote:**
> **
>
> Hi Mark,****
>
> ****
>
> NNBench is a namenode load test. Output of the test is the set of
> performance numbers, like transactions per second, average latency of
> operations, etc.****
>
> ****
>
> What do you mean by trying to create a more efficient namenode? What
> dimension are you trying to optimize? Depending on this, people on this
> list might be able to guide you to something more appropriate. ****
>
> ****
>
> Hope this helps,****
>
> Ivan ****
>
> ****
>
> *From:* Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
> *Sent:* Wednesday, June 05, 2013 3:17 PM
> *To:* Hadoop User
> *Subject:* How to test the performance of NN?****
>
> ****
>
> Hi,
>
> I am trying to create a more efficient namenode, and for that I need to
> the standard distribution, and then compare it to my version.
>
> Which benchmark should I run? I am doing nnbench, but it is not telling me
> anything about performance, only about potential failures.
>
> Thank you.****
>
> Sincerely,
> Mark****
>
> ** **
>
RE: How to test the performance of NN?
Posted by Ivan Mitic <iv...@microsoft.com>.
Hi Mark,
Your NNBench output does not look OK to me. Below is an example NNBench cmd line I used some time ago and its output.
Btw, what I found useful in testing the disk IO performance for namenode operations (others might find this unusual) is the TestEditLog unittest. I would modify the test as appropriate and run the simulation. This was a quick micro-benchmark I used to compare a couple of devices. Note that you'll have to modify the test in trunk and remove the line with EditLogFileOutputStream.setShouldSkipFsyncForTesting(), otherwise you're not testing the real thing.
hadoop.cmd jar hadoop-test-1.1.0-SNAPSHOT.jar nnbench -operation create_write -maps 32 -reduces 1 -blockSize 1 -bytesToWrite 20 -bytesPerChecksum 1 -numberOfFiles 500 -replicationFactorPerFile 1
-------------- NNBench -------------- :
Version: NameNode Benchmark 0.4
Date & time: 2012-12-18 07:29:20,832
Test Operation: create_write
Start time: 2012-12-18 07:25:36,111
Maps to run: 32
Reduces to run: 1
Block Size (bytes): 1
Bytes to write: 20
Bytes per checksum: 1
Number of files: 500
Replication factor: 1
Successful file operations: 16000
# maps that missed the barrier: 0
# exceptions: 0
TPS: Create/Write/Close: 153
Avg exec time (ms): Create/Write/Close: 301.6148125
Avg Lat (ms): Create/Write: 68.25625
Avg Lat (ms): Close: 233.245
RAW DATA: AL Total #1: 1092100
RAW DATA: AL Total #2: 3731920
RAW DATA: TPS Total (ms): 4825837
RAW DATA: Longest Map Time (ms): 208692.0
RAW DATA: Late maps: 0
RAW DATA: # of exceptions: 0
Hope this helps,
Ivan
From: Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
Sent: Wednesday, June 05, 2013 5:16 PM
To: Hadoop User
Subject: Re: How to test the performance of NN?
Hi, Ivan,
thank you for your willingness to help. First, what do I mean by a more efficient NN? - I am experimenting with Kove<http://kove.com/> device, persistent memory. I want to do one of the two things:
1. Use it for FSDirectory, with the expectation of very fast writes, to see if this improves performance.
2. Use it for BlockMaps, not for performance, but for fault-tolerance.
To prepare for this, I want to run some benchmarks on the given install just to get experience running it. Then I will run it on a 3-node cluster without Kove, and then with Kove.
Here is what I got from nnbench, and I am trying to find what I should watch for to measure NN performance. Does this all make sense?
Thank you,
Mark
13/06/05 16:50:16 INFO hdfs.NNBench: -------------- NNBench -------------- :
13/06/05 16:50:16 INFO hdfs.NNBench: Version: NameNode Benchmark 0.4
13/06/05 16:50:16 INFO hdfs.NNBench: Date & time: 2013-06-05 16:50:16,412
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: Test Operation: open_read
13/06/05 16:50:16 INFO hdfs.NNBench: Start time: 2013-06-05 16:48:34,792
13/06/05 16:50:16 INFO hdfs.NNBench: Maps to run: 12
13/06/05 16:50:16 INFO hdfs.NNBench: Reduces to run: 6
13/06/05 16:50:16 INFO hdfs.NNBench: Block Size (bytes): 1
13/06/05 16:50:16 INFO hdfs.NNBench: Bytes to write: 0
13/06/05 16:50:16 INFO hdfs.NNBench: Bytes per checksum: 1
13/06/05 16:50:16 INFO hdfs.NNBench: Number of files: 1000
13/06/05 16:50:16 INFO hdfs.NNBench: Replication factor: 3
13/06/05 16:50:16 INFO hdfs.NNBench: Successful file operations: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: # maps that missed the barrier: 0
13/06/05 16:50:16 INFO hdfs.NNBench: # exceptions: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: TPS: Open/Read: 0
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Exec time (ms): Open/Read: 0.0
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms): Open: Infinity
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms): Read: NaN
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total #1: 4665
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total #2: 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: TPS Total (ms): 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Longest Map Time (ms): 0.0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Late maps: 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: # of exceptions: 0
On Wed, Jun 5, 2013 at 4:32 PM, Ivan Mitic <iv...@microsoft.com>> wrote:
Hi Mark,
NNBench is a namenode load test. Output of the test is the set of performance numbers, like transactions per second, average latency of operations, etc.
What do you mean by trying to create a more efficient namenode? What dimension are you trying to optimize? Depending on this, people on this list might be able to guide you to something more appropriate.
Hope this helps,
Ivan
From: Mark Kerzner [mailto:mark.kerzner@shmsoft.com<ma...@shmsoft.com>]
Sent: Wednesday, June 05, 2013 3:17 PM
To: Hadoop User
Subject: How to test the performance of NN?
Hi,
I am trying to create a more efficient namenode, and for that I need to the standard distribution, and then compare it to my version.
Which benchmark should I run? I am doing nnbench, but it is not telling me anything about performance, only about potential failures.
Thank you.
Sincerely,
Mark
RE: How to test the performance of NN?
Posted by Ivan Mitic <iv...@microsoft.com>.
Hi Mark,
Your NNBench output does not look OK to me. Below is an example NNBench cmd line I used some time ago and its output.
Btw, what I found useful in testing the disk IO performance for namenode operations (others might find this unusual) is the TestEditLog unittest. I would modify the test as appropriate and run the simulation. This was a quick micro-benchmark I used to compare a couple of devices. Note that you'll have to modify the test in trunk and remove the line with EditLogFileOutputStream.setShouldSkipFsyncForTesting(), otherwise you're not testing the real thing.
hadoop.cmd jar hadoop-test-1.1.0-SNAPSHOT.jar nnbench -operation create_write -maps 32 -reduces 1 -blockSize 1 -bytesToWrite 20 -bytesPerChecksum 1 -numberOfFiles 500 -replicationFactorPerFile 1
-------------- NNBench -------------- :
Version: NameNode Benchmark 0.4
Date & time: 2012-12-18 07:29:20,832
Test Operation: create_write
Start time: 2012-12-18 07:25:36,111
Maps to run: 32
Reduces to run: 1
Block Size (bytes): 1
Bytes to write: 20
Bytes per checksum: 1
Number of files: 500
Replication factor: 1
Successful file operations: 16000
# maps that missed the barrier: 0
# exceptions: 0
TPS: Create/Write/Close: 153
Avg exec time (ms): Create/Write/Close: 301.6148125
Avg Lat (ms): Create/Write: 68.25625
Avg Lat (ms): Close: 233.245
RAW DATA: AL Total #1: 1092100
RAW DATA: AL Total #2: 3731920
RAW DATA: TPS Total (ms): 4825837
RAW DATA: Longest Map Time (ms): 208692.0
RAW DATA: Late maps: 0
RAW DATA: # of exceptions: 0
Hope this helps,
Ivan
From: Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
Sent: Wednesday, June 05, 2013 5:16 PM
To: Hadoop User
Subject: Re: How to test the performance of NN?
Hi, Ivan,
thank you for your willingness to help. First, what do I mean by a more efficient NN? - I am experimenting with Kove<http://kove.com/> device, persistent memory. I want to do one of the two things:
1. Use it for FSDirectory, with the expectation of very fast writes, to see if this improves performance.
2. Use it for BlockMaps, not for performance, but for fault-tolerance.
To prepare for this, I want to run some benchmarks on the given install just to get experience running it. Then I will run it on a 3-node cluster without Kove, and then with Kove.
Here is what I got from nnbench, and I am trying to find what I should watch for to measure NN performance. Does this all make sense?
Thank you,
Mark
13/06/05 16:50:16 INFO hdfs.NNBench: -------------- NNBench -------------- :
13/06/05 16:50:16 INFO hdfs.NNBench: Version: NameNode Benchmark 0.4
13/06/05 16:50:16 INFO hdfs.NNBench: Date & time: 2013-06-05 16:50:16,412
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: Test Operation: open_read
13/06/05 16:50:16 INFO hdfs.NNBench: Start time: 2013-06-05 16:48:34,792
13/06/05 16:50:16 INFO hdfs.NNBench: Maps to run: 12
13/06/05 16:50:16 INFO hdfs.NNBench: Reduces to run: 6
13/06/05 16:50:16 INFO hdfs.NNBench: Block Size (bytes): 1
13/06/05 16:50:16 INFO hdfs.NNBench: Bytes to write: 0
13/06/05 16:50:16 INFO hdfs.NNBench: Bytes per checksum: 1
13/06/05 16:50:16 INFO hdfs.NNBench: Number of files: 1000
13/06/05 16:50:16 INFO hdfs.NNBench: Replication factor: 3
13/06/05 16:50:16 INFO hdfs.NNBench: Successful file operations: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: # maps that missed the barrier: 0
13/06/05 16:50:16 INFO hdfs.NNBench: # exceptions: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: TPS: Open/Read: 0
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Exec time (ms): Open/Read: 0.0
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms): Open: Infinity
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms): Read: NaN
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total #1: 4665
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total #2: 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: TPS Total (ms): 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Longest Map Time (ms): 0.0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Late maps: 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: # of exceptions: 0
On Wed, Jun 5, 2013 at 4:32 PM, Ivan Mitic <iv...@microsoft.com>> wrote:
Hi Mark,
NNBench is a namenode load test. Output of the test is the set of performance numbers, like transactions per second, average latency of operations, etc.
What do you mean by trying to create a more efficient namenode? What dimension are you trying to optimize? Depending on this, people on this list might be able to guide you to something more appropriate.
Hope this helps,
Ivan
From: Mark Kerzner [mailto:mark.kerzner@shmsoft.com<ma...@shmsoft.com>]
Sent: Wednesday, June 05, 2013 3:17 PM
To: Hadoop User
Subject: How to test the performance of NN?
Hi,
I am trying to create a more efficient namenode, and for that I need to the standard distribution, and then compare it to my version.
Which benchmark should I run? I am doing nnbench, but it is not telling me anything about performance, only about potential failures.
Thank you.
Sincerely,
Mark
RE: How to test the performance of NN?
Posted by Ivan Mitic <iv...@microsoft.com>.
Hi Mark,
Your NNBench output does not look OK to me. Below is an example NNBench cmd line I used some time ago and its output.
Btw, what I found useful in testing the disk IO performance for namenode operations (others might find this unusual) is the TestEditLog unittest. I would modify the test as appropriate and run the simulation. This was a quick micro-benchmark I used to compare a couple of devices. Note that you'll have to modify the test in trunk and remove the line with EditLogFileOutputStream.setShouldSkipFsyncForTesting(), otherwise you're not testing the real thing.
hadoop.cmd jar hadoop-test-1.1.0-SNAPSHOT.jar nnbench -operation create_write -maps 32 -reduces 1 -blockSize 1 -bytesToWrite 20 -bytesPerChecksum 1 -numberOfFiles 500 -replicationFactorPerFile 1
-------------- NNBench -------------- :
Version: NameNode Benchmark 0.4
Date & time: 2012-12-18 07:29:20,832
Test Operation: create_write
Start time: 2012-12-18 07:25:36,111
Maps to run: 32
Reduces to run: 1
Block Size (bytes): 1
Bytes to write: 20
Bytes per checksum: 1
Number of files: 500
Replication factor: 1
Successful file operations: 16000
# maps that missed the barrier: 0
# exceptions: 0
TPS: Create/Write/Close: 153
Avg exec time (ms): Create/Write/Close: 301.6148125
Avg Lat (ms): Create/Write: 68.25625
Avg Lat (ms): Close: 233.245
RAW DATA: AL Total #1: 1092100
RAW DATA: AL Total #2: 3731920
RAW DATA: TPS Total (ms): 4825837
RAW DATA: Longest Map Time (ms): 208692.0
RAW DATA: Late maps: 0
RAW DATA: # of exceptions: 0
Hope this helps,
Ivan
From: Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
Sent: Wednesday, June 05, 2013 5:16 PM
To: Hadoop User
Subject: Re: How to test the performance of NN?
Hi, Ivan,
thank you for your willingness to help. First, what do I mean by a more efficient NN? - I am experimenting with Kove<http://kove.com/> device, persistent memory. I want to do one of the two things:
1. Use it for FSDirectory, with the expectation of very fast writes, to see if this improves performance.
2. Use it for BlockMaps, not for performance, but for fault-tolerance.
To prepare for this, I want to run some benchmarks on the given install just to get experience running it. Then I will run it on a 3-node cluster without Kove, and then with Kove.
Here is what I got from nnbench, and I am trying to find what I should watch for to measure NN performance. Does this all make sense?
Thank you,
Mark
13/06/05 16:50:16 INFO hdfs.NNBench: -------------- NNBench -------------- :
13/06/05 16:50:16 INFO hdfs.NNBench: Version: NameNode Benchmark 0.4
13/06/05 16:50:16 INFO hdfs.NNBench: Date & time: 2013-06-05 16:50:16,412
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: Test Operation: open_read
13/06/05 16:50:16 INFO hdfs.NNBench: Start time: 2013-06-05 16:48:34,792
13/06/05 16:50:16 INFO hdfs.NNBench: Maps to run: 12
13/06/05 16:50:16 INFO hdfs.NNBench: Reduces to run: 6
13/06/05 16:50:16 INFO hdfs.NNBench: Block Size (bytes): 1
13/06/05 16:50:16 INFO hdfs.NNBench: Bytes to write: 0
13/06/05 16:50:16 INFO hdfs.NNBench: Bytes per checksum: 1
13/06/05 16:50:16 INFO hdfs.NNBench: Number of files: 1000
13/06/05 16:50:16 INFO hdfs.NNBench: Replication factor: 3
13/06/05 16:50:16 INFO hdfs.NNBench: Successful file operations: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: # maps that missed the barrier: 0
13/06/05 16:50:16 INFO hdfs.NNBench: # exceptions: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: TPS: Open/Read: 0
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Exec time (ms): Open/Read: 0.0
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms): Open: Infinity
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms): Read: NaN
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total #1: 4665
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total #2: 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: TPS Total (ms): 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Longest Map Time (ms): 0.0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Late maps: 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: # of exceptions: 0
On Wed, Jun 5, 2013 at 4:32 PM, Ivan Mitic <iv...@microsoft.com>> wrote:
Hi Mark,
NNBench is a namenode load test. Output of the test is the set of performance numbers, like transactions per second, average latency of operations, etc.
What do you mean by trying to create a more efficient namenode? What dimension are you trying to optimize? Depending on this, people on this list might be able to guide you to something more appropriate.
Hope this helps,
Ivan
From: Mark Kerzner [mailto:mark.kerzner@shmsoft.com<ma...@shmsoft.com>]
Sent: Wednesday, June 05, 2013 3:17 PM
To: Hadoop User
Subject: How to test the performance of NN?
Hi,
I am trying to create a more efficient namenode, and for that I need to the standard distribution, and then compare it to my version.
Which benchmark should I run? I am doing nnbench, but it is not telling me anything about performance, only about potential failures.
Thank you.
Sincerely,
Mark
RE: How to test the performance of NN?
Posted by Ivan Mitic <iv...@microsoft.com>.
Hi Mark,
Your NNBench output does not look OK to me. Below is an example NNBench cmd line I used some time ago and its output.
Btw, what I found useful in testing the disk IO performance for namenode operations (others might find this unusual) is the TestEditLog unittest. I would modify the test as appropriate and run the simulation. This was a quick micro-benchmark I used to compare a couple of devices. Note that you'll have to modify the test in trunk and remove the line with EditLogFileOutputStream.setShouldSkipFsyncForTesting(), otherwise you're not testing the real thing.
hadoop.cmd jar hadoop-test-1.1.0-SNAPSHOT.jar nnbench -operation create_write -maps 32 -reduces 1 -blockSize 1 -bytesToWrite 20 -bytesPerChecksum 1 -numberOfFiles 500 -replicationFactorPerFile 1
-------------- NNBench -------------- :
Version: NameNode Benchmark 0.4
Date & time: 2012-12-18 07:29:20,832
Test Operation: create_write
Start time: 2012-12-18 07:25:36,111
Maps to run: 32
Reduces to run: 1
Block Size (bytes): 1
Bytes to write: 20
Bytes per checksum: 1
Number of files: 500
Replication factor: 1
Successful file operations: 16000
# maps that missed the barrier: 0
# exceptions: 0
TPS: Create/Write/Close: 153
Avg exec time (ms): Create/Write/Close: 301.6148125
Avg Lat (ms): Create/Write: 68.25625
Avg Lat (ms): Close: 233.245
RAW DATA: AL Total #1: 1092100
RAW DATA: AL Total #2: 3731920
RAW DATA: TPS Total (ms): 4825837
RAW DATA: Longest Map Time (ms): 208692.0
RAW DATA: Late maps: 0
RAW DATA: # of exceptions: 0
Hope this helps,
Ivan
From: Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
Sent: Wednesday, June 05, 2013 5:16 PM
To: Hadoop User
Subject: Re: How to test the performance of NN?
Hi, Ivan,
thank you for your willingness to help. First, what do I mean by a more efficient NN? - I am experimenting with Kove<http://kove.com/> device, persistent memory. I want to do one of the two things:
1. Use it for FSDirectory, with the expectation of very fast writes, to see if this improves performance.
2. Use it for BlockMaps, not for performance, but for fault-tolerance.
To prepare for this, I want to run some benchmarks on the given install just to get experience running it. Then I will run it on a 3-node cluster without Kove, and then with Kove.
Here is what I got from nnbench, and I am trying to find what I should watch for to measure NN performance. Does this all make sense?
Thank you,
Mark
13/06/05 16:50:16 INFO hdfs.NNBench: -------------- NNBench -------------- :
13/06/05 16:50:16 INFO hdfs.NNBench: Version: NameNode Benchmark 0.4
13/06/05 16:50:16 INFO hdfs.NNBench: Date & time: 2013-06-05 16:50:16,412
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: Test Operation: open_read
13/06/05 16:50:16 INFO hdfs.NNBench: Start time: 2013-06-05 16:48:34,792
13/06/05 16:50:16 INFO hdfs.NNBench: Maps to run: 12
13/06/05 16:50:16 INFO hdfs.NNBench: Reduces to run: 6
13/06/05 16:50:16 INFO hdfs.NNBench: Block Size (bytes): 1
13/06/05 16:50:16 INFO hdfs.NNBench: Bytes to write: 0
13/06/05 16:50:16 INFO hdfs.NNBench: Bytes per checksum: 1
13/06/05 16:50:16 INFO hdfs.NNBench: Number of files: 1000
13/06/05 16:50:16 INFO hdfs.NNBench: Replication factor: 3
13/06/05 16:50:16 INFO hdfs.NNBench: Successful file operations: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: # maps that missed the barrier: 0
13/06/05 16:50:16 INFO hdfs.NNBench: # exceptions: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: TPS: Open/Read: 0
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Exec time (ms): Open/Read: 0.0
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms): Open: Infinity
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms): Read: NaN
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total #1: 4665
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total #2: 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: TPS Total (ms): 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Longest Map Time (ms): 0.0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Late maps: 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: # of exceptions: 0
On Wed, Jun 5, 2013 at 4:32 PM, Ivan Mitic <iv...@microsoft.com>> wrote:
Hi Mark,
NNBench is a namenode load test. Output of the test is the set of performance numbers, like transactions per second, average latency of operations, etc.
What do you mean by trying to create a more efficient namenode? What dimension are you trying to optimize? Depending on this, people on this list might be able to guide you to something more appropriate.
Hope this helps,
Ivan
From: Mark Kerzner [mailto:mark.kerzner@shmsoft.com<ma...@shmsoft.com>]
Sent: Wednesday, June 05, 2013 3:17 PM
To: Hadoop User
Subject: How to test the performance of NN?
Hi,
I am trying to create a more efficient namenode, and for that I need to the standard distribution, and then compare it to my version.
Which benchmark should I run? I am doing nnbench, but it is not telling me anything about performance, only about potential failures.
Thank you.
Sincerely,
Mark
Re: How to test the performance of NN?
Posted by Mark Kerzner <ma...@shmsoft.com>.
Hi, Ivan,
thank you for your willingness to help. First, what do I mean by a more
efficient NN? - I am experimenting with Kove <http://kove.com/> device,
persistent memory. I want to do one of the two things:
1. Use it for FSDirectory, with the expectation of very fast writes, to see
if this improves performance.
2. Use it for BlockMaps, not for performance, but for fault-tolerance.
To prepare for this, I want to run some benchmarks on the given install
just to get experience running it. Then I will run it on a 3-node cluster
without Kove, and then with Kove.
Here is what I got from nnbench, and I am trying to find what I should
watch for to measure NN performance. Does this all make sense?
Thank you,
Mark
13/06/05 16:50:16 INFO hdfs.NNBench: -------------- NNBench -------------- :
13/06/05 16:50:16 INFO hdfs.NNBench:
Version: NameNode Benchmark 0.4
13/06/05 16:50:16 INFO hdfs.NNBench: Date &
time: 2013-06-05 16:50:16,412
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: Test
Operation: open_read
13/06/05 16:50:16 INFO hdfs.NNBench: Start
time: 2013-06-05 16:48:34,792
13/06/05 16:50:16 INFO hdfs.NNBench: Maps to
run: 12
13/06/05 16:50:16 INFO hdfs.NNBench: Reduces to
run: 6
13/06/05 16:50:16 INFO hdfs.NNBench: Block Size
(bytes): 1
13/06/05 16:50:16 INFO hdfs.NNBench: Bytes to
write: 0
13/06/05 16:50:16 INFO hdfs.NNBench: Bytes per
checksum: 1
13/06/05 16:50:16 INFO hdfs.NNBench: Number of
files: 1000
13/06/05 16:50:16 INFO hdfs.NNBench: Replication
factor: 3
13/06/05 16:50:16 INFO hdfs.NNBench: Successful file
operations: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: # maps that missed the
barrier: 0
13/06/05 16:50:16 INFO hdfs.NNBench: #
exceptions: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: TPS:
Open/Read: 0
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Exec time (ms):
Open/Read: 0.0
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms):
Open: Infinity
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms): Read:
NaN
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total
#1: 4665
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total
#2: 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: TPS Total
(ms): 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Longest Map Time
(ms): 0.0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Late
maps: 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: # of
exceptions: 0
On Wed, Jun 5, 2013 at 4:32 PM, Ivan Mitic <iv...@microsoft.com> wrote:
> Hi Mark,****
>
> ** **
>
> NNBench is a namenode load test. Output of the test is the set of
> performance numbers, like transactions per second, average latency of
> operations, etc.****
>
> ** **
>
> What do you mean by trying to create a more efficient namenode? What
> dimension are you trying to optimize? Depending on this, people on this
> list might be able to guide you to something more appropriate. ****
>
> ** **
>
> Hope this helps,****
>
> Ivan ****
>
> ** **
>
> *From:* Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
> *Sent:* Wednesday, June 05, 2013 3:17 PM
> *To:* Hadoop User
> *Subject:* How to test the performance of NN?****
>
> ** **
>
> Hi,
>
> I am trying to create a more efficient namenode, and for that I need to
> the standard distribution, and then compare it to my version.
>
> Which benchmark should I run? I am doing nnbench, but it is not telling me
> anything about performance, only about potential failures.
>
> Thank you.****
>
> Sincerely,
> Mark****
>
Re: How to test the performance of NN?
Posted by Mark Kerzner <ma...@shmsoft.com>.
Hi, Ivan,
thank you for your willingness to help. First, what do I mean by a more
efficient NN? - I am experimenting with Kove <http://kove.com/> device,
persistent memory. I want to do one of the two things:
1. Use it for FSDirectory, with the expectation of very fast writes, to see
if this improves performance.
2. Use it for BlockMaps, not for performance, but for fault-tolerance.
To prepare for this, I want to run some benchmarks on the given install
just to get experience running it. Then I will run it on a 3-node cluster
without Kove, and then with Kove.
Here is what I got from nnbench, and I am trying to find what I should
watch for to measure NN performance. Does this all make sense?
Thank you,
Mark
13/06/05 16:50:16 INFO hdfs.NNBench: -------------- NNBench -------------- :
13/06/05 16:50:16 INFO hdfs.NNBench:
Version: NameNode Benchmark 0.4
13/06/05 16:50:16 INFO hdfs.NNBench: Date &
time: 2013-06-05 16:50:16,412
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: Test
Operation: open_read
13/06/05 16:50:16 INFO hdfs.NNBench: Start
time: 2013-06-05 16:48:34,792
13/06/05 16:50:16 INFO hdfs.NNBench: Maps to
run: 12
13/06/05 16:50:16 INFO hdfs.NNBench: Reduces to
run: 6
13/06/05 16:50:16 INFO hdfs.NNBench: Block Size
(bytes): 1
13/06/05 16:50:16 INFO hdfs.NNBench: Bytes to
write: 0
13/06/05 16:50:16 INFO hdfs.NNBench: Bytes per
checksum: 1
13/06/05 16:50:16 INFO hdfs.NNBench: Number of
files: 1000
13/06/05 16:50:16 INFO hdfs.NNBench: Replication
factor: 3
13/06/05 16:50:16 INFO hdfs.NNBench: Successful file
operations: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: # maps that missed the
barrier: 0
13/06/05 16:50:16 INFO hdfs.NNBench: #
exceptions: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: TPS:
Open/Read: 0
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Exec time (ms):
Open/Read: 0.0
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms):
Open: Infinity
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms): Read:
NaN
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total
#1: 4665
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total
#2: 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: TPS Total
(ms): 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Longest Map Time
(ms): 0.0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Late
maps: 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: # of
exceptions: 0
On Wed, Jun 5, 2013 at 4:32 PM, Ivan Mitic <iv...@microsoft.com> wrote:
> Hi Mark,****
>
> ** **
>
> NNBench is a namenode load test. Output of the test is the set of
> performance numbers, like transactions per second, average latency of
> operations, etc.****
>
> ** **
>
> What do you mean by trying to create a more efficient namenode? What
> dimension are you trying to optimize? Depending on this, people on this
> list might be able to guide you to something more appropriate. ****
>
> ** **
>
> Hope this helps,****
>
> Ivan ****
>
> ** **
>
> *From:* Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
> *Sent:* Wednesday, June 05, 2013 3:17 PM
> *To:* Hadoop User
> *Subject:* How to test the performance of NN?****
>
> ** **
>
> Hi,
>
> I am trying to create a more efficient namenode, and for that I need to
> the standard distribution, and then compare it to my version.
>
> Which benchmark should I run? I am doing nnbench, but it is not telling me
> anything about performance, only about potential failures.
>
> Thank you.****
>
> Sincerely,
> Mark****
>
Re: How to test the performance of NN?
Posted by Mark Kerzner <ma...@shmsoft.com>.
Hi, Ivan,
thank you for your willingness to help. First, what do I mean by a more
efficient NN? - I am experimenting with Kove <http://kove.com/> device,
persistent memory. I want to do one of the two things:
1. Use it for FSDirectory, with the expectation of very fast writes, to see
if this improves performance.
2. Use it for BlockMaps, not for performance, but for fault-tolerance.
To prepare for this, I want to run some benchmarks on the given install
just to get experience running it. Then I will run it on a 3-node cluster
without Kove, and then with Kove.
Here is what I got from nnbench, and I am trying to find what I should
watch for to measure NN performance. Does this all make sense?
Thank you,
Mark
13/06/05 16:50:16 INFO hdfs.NNBench: -------------- NNBench -------------- :
13/06/05 16:50:16 INFO hdfs.NNBench:
Version: NameNode Benchmark 0.4
13/06/05 16:50:16 INFO hdfs.NNBench: Date &
time: 2013-06-05 16:50:16,412
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: Test
Operation: open_read
13/06/05 16:50:16 INFO hdfs.NNBench: Start
time: 2013-06-05 16:48:34,792
13/06/05 16:50:16 INFO hdfs.NNBench: Maps to
run: 12
13/06/05 16:50:16 INFO hdfs.NNBench: Reduces to
run: 6
13/06/05 16:50:16 INFO hdfs.NNBench: Block Size
(bytes): 1
13/06/05 16:50:16 INFO hdfs.NNBench: Bytes to
write: 0
13/06/05 16:50:16 INFO hdfs.NNBench: Bytes per
checksum: 1
13/06/05 16:50:16 INFO hdfs.NNBench: Number of
files: 1000
13/06/05 16:50:16 INFO hdfs.NNBench: Replication
factor: 3
13/06/05 16:50:16 INFO hdfs.NNBench: Successful file
operations: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: # maps that missed the
barrier: 0
13/06/05 16:50:16 INFO hdfs.NNBench: #
exceptions: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: TPS:
Open/Read: 0
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Exec time (ms):
Open/Read: 0.0
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms):
Open: Infinity
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms): Read:
NaN
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total
#1: 4665
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total
#2: 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: TPS Total
(ms): 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Longest Map Time
(ms): 0.0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Late
maps: 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: # of
exceptions: 0
On Wed, Jun 5, 2013 at 4:32 PM, Ivan Mitic <iv...@microsoft.com> wrote:
> Hi Mark,****
>
> ** **
>
> NNBench is a namenode load test. Output of the test is the set of
> performance numbers, like transactions per second, average latency of
> operations, etc.****
>
> ** **
>
> What do you mean by trying to create a more efficient namenode? What
> dimension are you trying to optimize? Depending on this, people on this
> list might be able to guide you to something more appropriate. ****
>
> ** **
>
> Hope this helps,****
>
> Ivan ****
>
> ** **
>
> *From:* Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
> *Sent:* Wednesday, June 05, 2013 3:17 PM
> *To:* Hadoop User
> *Subject:* How to test the performance of NN?****
>
> ** **
>
> Hi,
>
> I am trying to create a more efficient namenode, and for that I need to
> the standard distribution, and then compare it to my version.
>
> Which benchmark should I run? I am doing nnbench, but it is not telling me
> anything about performance, only about potential failures.
>
> Thank you.****
>
> Sincerely,
> Mark****
>
Re: How to test the performance of NN?
Posted by Mark Kerzner <ma...@shmsoft.com>.
Hi, Ivan,
thank you for your willingness to help. First, what do I mean by a more
efficient NN? - I am experimenting with Kove <http://kove.com/> device,
persistent memory. I want to do one of the two things:
1. Use it for FSDirectory, with the expectation of very fast writes, to see
if this improves performance.
2. Use it for BlockMaps, not for performance, but for fault-tolerance.
To prepare for this, I want to run some benchmarks on the given install
just to get experience running it. Then I will run it on a 3-node cluster
without Kove, and then with Kove.
Here is what I got from nnbench, and I am trying to find what I should
watch for to measure NN performance. Does this all make sense?
Thank you,
Mark
13/06/05 16:50:16 INFO hdfs.NNBench: -------------- NNBench -------------- :
13/06/05 16:50:16 INFO hdfs.NNBench:
Version: NameNode Benchmark 0.4
13/06/05 16:50:16 INFO hdfs.NNBench: Date &
time: 2013-06-05 16:50:16,412
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: Test
Operation: open_read
13/06/05 16:50:16 INFO hdfs.NNBench: Start
time: 2013-06-05 16:48:34,792
13/06/05 16:50:16 INFO hdfs.NNBench: Maps to
run: 12
13/06/05 16:50:16 INFO hdfs.NNBench: Reduces to
run: 6
13/06/05 16:50:16 INFO hdfs.NNBench: Block Size
(bytes): 1
13/06/05 16:50:16 INFO hdfs.NNBench: Bytes to
write: 0
13/06/05 16:50:16 INFO hdfs.NNBench: Bytes per
checksum: 1
13/06/05 16:50:16 INFO hdfs.NNBench: Number of
files: 1000
13/06/05 16:50:16 INFO hdfs.NNBench: Replication
factor: 3
13/06/05 16:50:16 INFO hdfs.NNBench: Successful file
operations: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: # maps that missed the
barrier: 0
13/06/05 16:50:16 INFO hdfs.NNBench: #
exceptions: 0
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: TPS:
Open/Read: 0
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Exec time (ms):
Open/Read: 0.0
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms):
Open: Infinity
13/06/05 16:50:16 INFO hdfs.NNBench: Avg Lat (ms): Read:
NaN
13/06/05 16:50:16 INFO hdfs.NNBench:
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total
#1: 4665
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: AL Total
#2: 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: TPS Total
(ms): 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Longest Map Time
(ms): 0.0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: Late
maps: 0
13/06/05 16:50:16 INFO hdfs.NNBench: RAW DATA: # of
exceptions: 0
On Wed, Jun 5, 2013 at 4:32 PM, Ivan Mitic <iv...@microsoft.com> wrote:
> Hi Mark,****
>
> ** **
>
> NNBench is a namenode load test. Output of the test is the set of
> performance numbers, like transactions per second, average latency of
> operations, etc.****
>
> ** **
>
> What do you mean by trying to create a more efficient namenode? What
> dimension are you trying to optimize? Depending on this, people on this
> list might be able to guide you to something more appropriate. ****
>
> ** **
>
> Hope this helps,****
>
> Ivan ****
>
> ** **
>
> *From:* Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
> *Sent:* Wednesday, June 05, 2013 3:17 PM
> *To:* Hadoop User
> *Subject:* How to test the performance of NN?****
>
> ** **
>
> Hi,
>
> I am trying to create a more efficient namenode, and for that I need to
> the standard distribution, and then compare it to my version.
>
> Which benchmark should I run? I am doing nnbench, but it is not telling me
> anything about performance, only about potential failures.
>
> Thank you.****
>
> Sincerely,
> Mark****
>
RE: How to test the performance of NN?
Posted by Ivan Mitic <iv...@microsoft.com>.
Hi Mark,
NNBench is a namenode load test. Output of the test is the set of performance numbers, like transactions per second, average latency of operations, etc.
What do you mean by trying to create a more efficient namenode? What dimension are you trying to optimize? Depending on this, people on this list might be able to guide you to something more appropriate.
Hope this helps,
Ivan
From: Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
Sent: Wednesday, June 05, 2013 3:17 PM
To: Hadoop User
Subject: How to test the performance of NN?
Hi,
I am trying to create a more efficient namenode, and for that I need to the standard distribution, and then compare it to my version.
Which benchmark should I run? I am doing nnbench, but it is not telling me anything about performance, only about potential failures.
Thank you.
Sincerely,
Mark
Re: How to test the performance of NN?
Posted by Suresh Srinivas <su...@hortonworks.com>.
What do you mean by it is not telling me any thing about performance? Also
I do not understand the part, "only about potential failures.". Can you add
more details.
nnbench is the best microbenchmark for nn performance test.
On Wed, Jun 5, 2013 at 3:17 PM, Mark Kerzner <ma...@shmsoft.com>wrote:
> Hi,
>
> I am trying to create a more efficient namenode, and for that I need to
> the standard distribution, and then compare it to my version.
>
> Which benchmark should I run? I am doing nnbench, but it is not telling me
> anything about performance, only about potential failures.
>
> Thank you.
> Sincerely,
> Mark
>
--
http://hortonworks.com/download/
Re: How to test the performance of NN?
Posted by Suresh Srinivas <su...@hortonworks.com>.
What do you mean by it is not telling me any thing about performance? Also
I do not understand the part, "only about potential failures.". Can you add
more details.
nnbench is the best microbenchmark for nn performance test.
On Wed, Jun 5, 2013 at 3:17 PM, Mark Kerzner <ma...@shmsoft.com>wrote:
> Hi,
>
> I am trying to create a more efficient namenode, and for that I need to
> the standard distribution, and then compare it to my version.
>
> Which benchmark should I run? I am doing nnbench, but it is not telling me
> anything about performance, only about potential failures.
>
> Thank you.
> Sincerely,
> Mark
>
--
http://hortonworks.com/download/
Re: How to test the performance of NN?
Posted by Suresh Srinivas <su...@hortonworks.com>.
What do you mean by it is not telling me any thing about performance? Also
I do not understand the part, "only about potential failures.". Can you add
more details.
nnbench is the best microbenchmark for nn performance test.
On Wed, Jun 5, 2013 at 3:17 PM, Mark Kerzner <ma...@shmsoft.com>wrote:
> Hi,
>
> I am trying to create a more efficient namenode, and for that I need to
> the standard distribution, and then compare it to my version.
>
> Which benchmark should I run? I am doing nnbench, but it is not telling me
> anything about performance, only about potential failures.
>
> Thank you.
> Sincerely,
> Mark
>
--
http://hortonworks.com/download/
RE: How to test the performance of NN?
Posted by Ivan Mitic <iv...@microsoft.com>.
Hi Mark,
NNBench is a namenode load test. Output of the test is the set of performance numbers, like transactions per second, average latency of operations, etc.
What do you mean by trying to create a more efficient namenode? What dimension are you trying to optimize? Depending on this, people on this list might be able to guide you to something more appropriate.
Hope this helps,
Ivan
From: Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
Sent: Wednesday, June 05, 2013 3:17 PM
To: Hadoop User
Subject: How to test the performance of NN?
Hi,
I am trying to create a more efficient namenode, and for that I need to the standard distribution, and then compare it to my version.
Which benchmark should I run? I am doing nnbench, but it is not telling me anything about performance, only about potential failures.
Thank you.
Sincerely,
Mark
RE: How to test the performance of NN?
Posted by Ivan Mitic <iv...@microsoft.com>.
Hi Mark,
NNBench is a namenode load test. Output of the test is the set of performance numbers, like transactions per second, average latency of operations, etc.
What do you mean by trying to create a more efficient namenode? What dimension are you trying to optimize? Depending on this, people on this list might be able to guide you to something more appropriate.
Hope this helps,
Ivan
From: Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
Sent: Wednesday, June 05, 2013 3:17 PM
To: Hadoop User
Subject: How to test the performance of NN?
Hi,
I am trying to create a more efficient namenode, and for that I need to the standard distribution, and then compare it to my version.
Which benchmark should I run? I am doing nnbench, but it is not telling me anything about performance, only about potential failures.
Thank you.
Sincerely,
Mark
RE: How to test the performance of NN?
Posted by Ivan Mitic <iv...@microsoft.com>.
Hi Mark,
NNBench is a namenode load test. Output of the test is the set of performance numbers, like transactions per second, average latency of operations, etc.
What do you mean by trying to create a more efficient namenode? What dimension are you trying to optimize? Depending on this, people on this list might be able to guide you to something more appropriate.
Hope this helps,
Ivan
From: Mark Kerzner [mailto:mark.kerzner@shmsoft.com]
Sent: Wednesday, June 05, 2013 3:17 PM
To: Hadoop User
Subject: How to test the performance of NN?
Hi,
I am trying to create a more efficient namenode, and for that I need to the standard distribution, and then compare it to my version.
Which benchmark should I run? I am doing nnbench, but it is not telling me anything about performance, only about potential failures.
Thank you.
Sincerely,
Mark
Re: How to test the performance of NN?
Posted by Suresh Srinivas <su...@hortonworks.com>.
What do you mean by it is not telling me any thing about performance? Also
I do not understand the part, "only about potential failures.". Can you add
more details.
nnbench is the best microbenchmark for nn performance test.
On Wed, Jun 5, 2013 at 3:17 PM, Mark Kerzner <ma...@shmsoft.com>wrote:
> Hi,
>
> I am trying to create a more efficient namenode, and for that I need to
> the standard distribution, and then compare it to my version.
>
> Which benchmark should I run? I am doing nnbench, but it is not telling me
> anything about performance, only about potential failures.
>
> Thank you.
> Sincerely,
> Mark
>
--
http://hortonworks.com/download/