You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Nigel Daley (JIRA)" <ji...@apache.org> on 2006/11/30 00:55:21 UTC
[jira] Created: (HADOOP-763) NameNode benchmark using mapred is
insufficient
NameNode benchmark using mapred is insufficient
-----------------------------------------------
Key: HADOOP-763
URL: http://issues.apache.org/jira/browse/HADOOP-763
Project: Hadoop
Issue Type: Improvement
Components: test
Affects Versions: 0.8.0
Reporter: Nigel Daley
Assigned To: Nigel Daley
Priority: Minor
The current namenode benchmark (org.apache.hadoop.examples.NNBench) uses map/reduce to distribute a load on the namenode. For the purposes of loading the namenode, this model gives insufficient control over job start and failure recovery. I propose the namenode benchmark be re-written to use slaves.sh directly to execute the namenode benchmark. The benchmark should also give finer control over the operations executed and the timings reported.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-763) NameNode benchmark using mapred is
insufficient
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-763?page=comments#action_12454771 ]
Hadoop QA commented on HADOOP-763:
----------------------------------
+1, http://issues.apache.org/jira/secure/attachment/12346076/HADOOP-763.patch applied and successfully tested against trunk revision 480686
> NameNode benchmark using mapred is insufficient
> -----------------------------------------------
>
> Key: HADOOP-763
> URL: http://issues.apache.org/jira/browse/HADOOP-763
> Project: Hadoop
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.8.0
> Reporter: Nigel Daley
> Assigned To: Nigel Daley
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HADOOP-763.patch
>
>
> The current namenode benchmark (org.apache.hadoop.examples.NNBench) uses map/reduce to distribute a load on the namenode. For the purposes of loading the namenode, this model gives insufficient control over job start and failure recovery. I propose the namenode benchmark be re-written to use slaves.sh directly to execute the namenode benchmark. The benchmark should also give finer control over the operations executed and the timings reported.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-763) NameNode benchmark using mapred is
insufficient
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-763?page=all ]
Nigel Daley updated HADOOP-763:
-------------------------------
Attachment: HADOOP-763.patch
Milind, others,
Please review new namenode benchmark.
> NameNode benchmark using mapred is insufficient
> -----------------------------------------------
>
> Key: HADOOP-763
> URL: http://issues.apache.org/jira/browse/HADOOP-763
> Project: Hadoop
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.8.0
> Reporter: Nigel Daley
> Assigned To: Nigel Daley
> Priority: Minor
> Attachments: HADOOP-763.patch
>
>
> The current namenode benchmark (org.apache.hadoop.examples.NNBench) uses map/reduce to distribute a load on the namenode. For the purposes of loading the namenode, this model gives insufficient control over job start and failure recovery. I propose the namenode benchmark be re-written to use slaves.sh directly to execute the namenode benchmark. The benchmark should also give finer control over the operations executed and the timings reported.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-763) NameNode benchmark using mapred is
insufficient
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-763?page=comments#action_12454769 ]
Nigel Daley commented on HADOOP-763:
------------------------------------
Elaborating on why running the namenode benchmark as a map/reduce is problematic, if a map fails, you either don't want it to rerun (not currently possible) or you want to detect a rerun map and disregard its output (not easily possible).
> NameNode benchmark using mapred is insufficient
> -----------------------------------------------
>
> Key: HADOOP-763
> URL: http://issues.apache.org/jira/browse/HADOOP-763
> Project: Hadoop
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.8.0
> Reporter: Nigel Daley
> Assigned To: Nigel Daley
> Priority: Minor
> Attachments: HADOOP-763.patch
>
>
> The current namenode benchmark (org.apache.hadoop.examples.NNBench) uses map/reduce to distribute a load on the namenode. For the purposes of loading the namenode, this model gives insufficient control over job start and failure recovery. I propose the namenode benchmark be re-written to use slaves.sh directly to execute the namenode benchmark. The benchmark should also give finer control over the operations executed and the timings reported.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-763) NameNode benchmark using mapred is
insufficient
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-763?page=all ]
Doug Cutting updated HADOOP-763:
--------------------------------
Status: Resolved (was: Patch Available)
Fix Version/s: 0.10.0
Resolution: Fixed
I just committed this. Thanks, Nigel!
> NameNode benchmark using mapred is insufficient
> -----------------------------------------------
>
> Key: HADOOP-763
> URL: http://issues.apache.org/jira/browse/HADOOP-763
> Project: Hadoop
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.8.0
> Reporter: Nigel Daley
> Assigned To: Nigel Daley
> Priority: Minor
> Fix For: 0.10.0
>
> Attachments: HADOOP-763.patch
>
>
> The current namenode benchmark (org.apache.hadoop.examples.NNBench) uses map/reduce to distribute a load on the namenode. For the purposes of loading the namenode, this model gives insufficient control over job start and failure recovery. I propose the namenode benchmark be re-written to use slaves.sh directly to execute the namenode benchmark. The benchmark should also give finer control over the operations executed and the timings reported.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-763) NameNode benchmark using mapred is
insufficient
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-763?page=comments#action_12454503 ]
Milind Bhandarkar commented on HADOOP-763:
------------------------------------------
Looks good.
+1
> NameNode benchmark using mapred is insufficient
> -----------------------------------------------
>
> Key: HADOOP-763
> URL: http://issues.apache.org/jira/browse/HADOOP-763
> Project: Hadoop
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.8.0
> Reporter: Nigel Daley
> Assigned To: Nigel Daley
> Priority: Minor
> Attachments: HADOOP-763.patch
>
>
> The current namenode benchmark (org.apache.hadoop.examples.NNBench) uses map/reduce to distribute a load on the namenode. For the purposes of loading the namenode, this model gives insufficient control over job start and failure recovery. I propose the namenode benchmark be re-written to use slaves.sh directly to execute the namenode benchmark. The benchmark should also give finer control over the operations executed and the timings reported.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-763) NameNode benchmark using mapred is
insufficient
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-763?page=all ]
Nigel Daley updated HADOOP-763:
-------------------------------
Status: Patch Available (was: Open)
Fix Version/s: 0.9.0
> NameNode benchmark using mapred is insufficient
> -----------------------------------------------
>
> Key: HADOOP-763
> URL: http://issues.apache.org/jira/browse/HADOOP-763
> Project: Hadoop
> Issue Type: Improvement
> Components: test
> Affects Versions: 0.8.0
> Reporter: Nigel Daley
> Assigned To: Nigel Daley
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HADOOP-763.patch
>
>
> The current namenode benchmark (org.apache.hadoop.examples.NNBench) uses map/reduce to distribute a load on the namenode. For the purposes of loading the namenode, this model gives insufficient control over job start and failure recovery. I propose the namenode benchmark be re-written to use slaves.sh directly to execute the namenode benchmark. The benchmark should also give finer control over the operations executed and the timings reported.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira