You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Mukund Madhugiri (JIRA)" <ji...@apache.org> on 2007/10/05 23:39:50 UTC
[jira] Created: (HADOOP-2000) Re-write NNBench to use MapReduce
Re-write NNBench to use MapReduce
---------------------------------
Key: HADOOP-2000
URL: https://issues.apache.org/jira/browse/HADOOP-2000
Project: Hadoop
Issue Type: Test
Components: test
Affects Versions: 0.15.0
Reporter: Mukund Madhugiri
Assignee: Mukund Madhugiri
The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
1. Transactions per second
2. Average latency
for these operations
- Create and Close file
- Open file
- Rename file
- Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12539756 ]
Konstantin Shvachko commented on HADOOP-2000:
---------------------------------------------
I think we should have the following two measures:
# average operation execution time = Number_of_operations / Sum(t_i)
where t_i is the time of each map task.
# Cluster throughput = Number_of_operations / Max(t_i).
where Max(t_i) is the longest map task.
(1) is what is currently called TPS in the patch.
(2) is a new measure, which measures how many operations per second the cluster can perform as a whole.
I think 2 might be a good candidate for the real throughput, because completion
of the longest map means that all other maps are done by that time too.
So this should indicate the actual end of the map stage of the job.
The only problem I see here is that this works only under the assumption that all maps
run in parallel, which is not always true, especially when failed maps have to be reptried.
So this is in a sense an ideal cluster throughput.
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2000:
-------------------------------------
Attachment: HADOOP-2000.patch
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Attachments: HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2000:
-------------------------------------
Status: Open (was: Patch Available)
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550418 ]
Hudson commented on HADOOP-2000:
--------------------------------
Integrated in Hadoop-Nightly #329 (See [http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/329/])
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2000:
-------------------------------------
Status: Patch Available (was: Open)
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2000:
-------------------------------------
Status: Patch Available (was: Open)
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535291 ]
Nigel Daley commented on HADOOP-2000:
-------------------------------------
Looks good. A few comments:
* import explicit classes, rather than java.io.*
* I'm not sure it makes sense to turn this into a junit test. As it stands, it won't get run by the test target because the class doesn't start with the word "Test"
* usage message is missing -bytesToWrite and -operation
* the operation "all" doesn't make sense
* Test Inputs log message missing some parameters
* the barrier() call should go into the mappers map method right before it's ready to start creating/opening/deleting/renaming and this should be documented so that user set a startTime that is far enough in the future
* barrier() should throw an exception if the startTime has already passed
* the map should only run 1 operation
* speculation should be turned off explicitly
* "Create a file" comment wrong in doRenameOp and doDeleteOp methods
* doCreateCloseOp should really be doCreateWriteOp and it should create the file with the appropriate blockSize and replicationFactor and then write the appropriate number of bytes (which may be zero)
* should there be exception handling around the various file io calls?
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2000:
-------------------------------------
Status: Patch Available (was: Open)
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-2000:
--------------------------------
Status: Open (was: Patch Available)
Great! Almost there.
* 30 second barrier default I think is too low. I'd suggest at 1 or 2 minutes.
* remove debug output
* optional: validateInputs() should check for negative parameter inputs that don't make sense
* optional: parseInputs() should output startTime in a format that is the default used in NN and DN logs
* in barrier(), "if (currentTime <= startTime)" would be clearer as "if (sleepTime > 0)"
* totalTimeTPS and startTimeTPS are really TPmS
* f: and s: are never used so that code can be removed from the reducer
* in map(), the openRead, rename, and delete cases should not call doCreateWriteOp as this screws up the point of the barrier; the assumption is that the createWrite operation is done before the other operations -- perhaps this needs to be in the usage message and javadoc
* rename maxFailedExceptions to MAX_OPERATION_EXCEPTIONS
* shouldn't replication factor be a short throughout?
* shouldn't openRead optionally read the file and record the time for reading?
* I think number of files need to be sent to the reducer since it may be less than the desired if the map is late or there are too many exceptioins; this should correct the value that is currently used in the analyzeResults() method
* remove tps1and2 and just multiply tps1 (which you then may want to rename) in the "If (operation.equals(OP_CREATE_WRITE))"
* document the forumulas in the analyzeResult() method
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2000:
-------------------------------------
Status: Patch Available (was: Open)
Updated the patch to address review feedback
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2000:
-------------------------------------
Attachment: HADOOP-2000.patch
Updated patch addressing review comments, including optional items
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2000:
-------------------------------------
Attachment: HADOOP-2000.patch
Updated patch addressing review comments
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2000:
-------------------------------------
Attachment: HADOOP-2000.patch
Updated patch addressing comments from Konstantin
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2000:
-------------------------------------
Attachment: HADOOP-2000.patch
Updated code to adhere to coding guidelines
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550173 ]
Hadoop QA commented on HADOOP-2000:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12371372/HADOOP-2000.patch
against trunk revision r603000.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1307/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1307/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1307/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1307/console
This message is automatically generated.
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2000:
-------------------------------------
Status: Patch Available (was: Open)
Attaching patch for code review
Here is a sample run output:
---------------------------------------------
bin/hadoop jar hadoop-0.15.0-dev-test.jar nnbench -maps 1 -reduces 1 -startTime `date +"%s"` -blockSize 1 -bytesPerChecksum 1 -replicationFactorPerFile 1 -numberOfFiles 100
NameNode Benchmark 0.4
07/10/09 18:14:09 INFO dfs.NNBench: Test Inputs:
07/10/09 18:14:09 INFO dfs.NNBench: Test Operation: all
07/10/09 18:14:09 INFO dfs.NNBench: Number of maps: 1
07/10/09 18:14:09 INFO dfs.NNBench: Number of reduces: 1
07/10/09 18:14:09 INFO dfs.NNBench: Block Size: 1
07/10/09 18:14:09 INFO dfs.NNBench: Bytes per checksum: 1
07/10/09 18:14:09 INFO dfs.NNBench: Number of files: 100
07/10/09 18:14:09 INFO dfs.NNBench: Replication factor: 1
07/10/09 18:14:10 INFO dfs.NNBench: Creating control files
07/10/09 18:14:10 INFO mapred.FileInputFormat: Total input paths to process : 1
07/10/09 18:14:10 INFO mapred.JobClient: Running job: job_200710032050_0199
07/10/09 18:14:11 INFO mapred.JobClient: map 0% reduce 0%
07/10/09 18:14:17 INFO mapred.JobClient: map 100% reduce 0%
07/10/09 18:14:18 INFO mapred.JobClient: map 100% reduce 100%
07/10/09 18:14:19 INFO mapred.JobClient: Job complete: job_200710032050_0199
07/10/09 18:14:19 INFO mapred.JobClient: Counters: 10
07/10/09 18:14:19 INFO mapred.JobClient: Job Counters
07/10/09 18:14:19 INFO mapred.JobClient: Launched map tasks=1
07/10/09 18:14:19 INFO mapred.JobClient: Launched reduce tasks=1
07/10/09 18:14:19 INFO mapred.JobClient: Data-local map tasks=1
07/10/09 18:14:19 INFO mapred.JobClient: Map-Reduce Framework
07/10/09 18:14:19 INFO mapred.JobClient: Map input records=1
07/10/09 18:14:19 INFO mapred.JobClient: Map output records=5
07/10/09 18:14:19 INFO mapred.JobClient: Map input bytes=39
07/10/09 18:14:19 INFO mapred.JobClient: Map output bytes=90
07/10/09 18:14:19 INFO mapred.JobClient: Reduce input groups=5
07/10/09 18:14:19 INFO mapred.JobClient: Reduce input records=5
07/10/09 18:14:19 INFO mapred.JobClient: Reduce output records=5
07/10/09 18:14:19 INFO dfs.NNBench: ------ NNBench ------ :
07/10/09 18:14:19 INFO dfs.NNBench: Version: NameNode Benchmark 0.4
07/10/09 18:14:19 INFO dfs.NNBench: Date & time: Tue Oct 09 18:14:19 UTC 2007
07/10/09 18:14:19 INFO dfs.NNBench:
07/10/09 18:14:19 INFO dfs.NNBench: Test Operation: all
07/10/09 18:14:19 INFO dfs.NNBench: Maps: 1
07/10/09 18:14:19 INFO dfs.NNBench: Reduces: 1
07/10/09 18:14:19 INFO dfs.NNBench: Block Size (bytes): 1
07/10/09 18:14:19 INFO dfs.NNBench: Bytes to write: 1
07/10/09 18:14:19 INFO dfs.NNBench: Bytes per checksum: 1
07/10/09 18:14:19 INFO dfs.NNBench: Number of files: 100
07/10/09 18:14:19 INFO dfs.NNBench: Replication factor: 1
07/10/09 18:14:19 INFO dfs.NNBench:
07/10/09 18:14:19 INFO dfs.NNBench: TPS: Create and Close: 72
07/10/09 18:14:19 INFO dfs.NNBench: TPS: Open (Read): 2777
07/10/09 18:14:19 INFO dfs.NNBench: TPS: Rename: 800
07/10/09 18:14:19 INFO dfs.NNBench: TPS: Delete: 7142
07/10/09 18:14:19 INFO dfs.NNBench:
07/10/09 18:14:19 INFO dfs.NNBench: Avg Lat (ms): Create: 1.35
07/10/09 18:14:19 INFO dfs.NNBench: Avg Lat (ms): Close: 26.28
07/10/09 18:14:19 INFO dfs.NNBench: Avg Lat (ms): Open (Read): 0.36
07/10/09 18:14:19 INFO dfs.NNBench: Avg Lat (ms): Rename: 1.25
07/10/09 18:14:19 INFO dfs.NNBench: Avg Lat (ms): Delete: 0.14
---------------------------------------------
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12538547 ]
Hadoop QA commented on HADOOP-2000:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12368439/HADOOP-2000.patch
against trunk revision r589576.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1022/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1022/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1022/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1022/console
This message is automatically generated.
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12549273 ]
Konstantin Shvachko commented on HADOOP-2000:
---------------------------------------------
A few minor comments:
- Redundant imports:
{code}
import java.text.DateFormat;
import org.apache.hadoop.io.UTF8;
import org.apache.hadoop.mapred.Reducer;
{code}
- Usage should specify mandatory options, like -operation, and optional ones.
- I get ArrayIndexOutOfBoundsException if I run any of the
{code}
NNBench -operation
NNBench -bytesToWrite
{code}
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2000:
-------------------------------------
Attachment: HADOOP-2000.patch
Uploading new patch to address comments from Konstantin
Three measurements are now being calculated:
1. Cluster Transactions per second (TPS) for an operation.
2. Average execution time in ms for an operation.
3. Average latency in ms for a transaction.
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2000:
-------------------------------------
Attachment: HADOOP-2000.patch
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2000:
-------------------------------------
Status: Patch Available (was: Open)
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12539565 ]
Konstantin Shvachko commented on HADOOP-2000:
---------------------------------------------
# redundant imports
import java.text.DateFormat;
import org.apache.hadoop.mapred.Reducer;
# variable name in NNBenchMapper.map() is never used.
# Typo
{code}
// Set user-dfined parameters,
{code}
# Printing TPS calculating TPmS. Should be the same:
{code}
" RAW DATA: TPS Total : " + totalTimeTPmS,
{code}
# double totalTimeTPS is confusing, since it is in fact TPS, not time according to the formula and the comments
# I am not happy with the whole concept of transactions per second.
So you measure total execution time of each map (t_i) and then divide Number_of_files / Sum(t_i).
But the Sum(t_i) is not the right time, because maps are running in parallel,
so in order to obtain the true TPS you need to time the start and the end of +*all*+ maps
rather than the start and the end of +*individual*+ maps.
But it is hard to get the exact starting and ending times of the job's map stage.
Your proposed TPS measures the # of transactions per second of a single client under a certain load on the cluster.
This is not completely unreasonable, but does not say much as a benchmark result imo.
I mean it is quite clear that if the cluster bears more load the clients run slower.
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-2000:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Mukund!
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2000:
-------------------------------------
Status: Open (was: Patch Available)
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2000:
-------------------------------------
Status: Open (was: Patch Available)
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12544128 ]
Hadoop QA commented on HADOOP-2000:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12369916/HADOOP-2000.patch
against trunk revision r596835.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1131/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1131/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1131/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1131/console
This message is automatically generated.
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12533474 ]
Hadoop QA commented on HADOOP-2000:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12367371/HADOOP-2000.patch
against trunk revision r583037.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/909/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/909/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/909/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/909/console
This message is automatically generated.
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Attachments: HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12538586 ]
Hadoop QA commented on HADOOP-2000:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12368616/HADOOP-2000.patch
against trunk revision r589576.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1023/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1023/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1023/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1023/console
This message is automatically generated.
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12550178 ]
Konstantin Shvachko commented on HADOOP-2000:
---------------------------------------------
+1
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2000:
-------------------------------------
Status: Patch Available (was: Open)
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12537905 ]
Hadoop QA commented on HADOOP-2000:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12368439/HADOOP-2000.patch
against trunk revision r588588.
@author +1. The patch does not contain any @author tags.
javadoc -1. The javadoc tool appears to have generated messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs -1. The patch appears to introduce new Findbugs warnings.
core tests -1. The patch failed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1009/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1009/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1009/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1009/console
This message is automatically generated.
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-2000:
-------------------------------------
Status: Open (was: Patch Available)
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2000) Re-write NNBench to use MapReduce
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-2000:
--------------------------------
Fix Version/s: 0.16.0
Status: Open (was: Patch Available)
> Re-write NNBench to use MapReduce
> ---------------------------------
>
> Key: HADOOP-2000
> URL: https://issues.apache.org/jira/browse/HADOOP-2000
> Project: Hadoop
> Issue Type: Test
> Components: test
> Affects Versions: 0.15.0
> Reporter: Mukund Madhugiri
> Assignee: Mukund Madhugiri
> Fix For: 0.16.0
>
> Attachments: HADOOP-2000.patch, HADOOP-2000.patch
>
>
> The proposal is to re-write the NNBench benchmark/test to measure Namenode operations using MapReduce. Two buckets of measurements will be done:
> 1. Transactions per second
> 2. Average latency
> for these operations
> - Create and Close file
> - Open file
> - Rename file
> - Delete file
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.