You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hemanth Yamijala (JIRA)" <ji...@apache.org> on 2008/07/04 13:55:31 UTC
[jira] Created: (HADOOP-3695) [HOD] Have an ability to run multiple
slaves per node
[HOD] Have an ability to run multiple slaves per node
-----------------------------------------------------
Key: HADOOP-3695
URL: https://issues.apache.org/jira/browse/HADOOP-3695
Project: Hadoop Core
Issue Type: New Feature
Components: contrib/hod
Reporter: Hemanth Yamijala
Assignee: Vinod Kumar Vavilapalli
Fix For: 0.19.0
Currently HOD launches at most one slave per node. For purposes of testing a large number of slaves on much fewer resources - for e.g. testing scalability of clusters, it will be useful if it can provision multiple slaves per node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3695) [HOD] Have an ability to run multiple
slaves per node
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-3695:
-------------------------------------
Attachment: patch_multiple_workers_4.txt
This patch modifies documentation in Vinod's last patch to add that this feature is basically for test/simulation purposes, and the workers need to be configured to use a proportional fraction of the resources on the node.
> [HOD] Have an ability to run multiple slaves per node
> -----------------------------------------------------
>
> Key: HADOOP-3695
> URL: https://issues.apache.org/jira/browse/HADOOP-3695
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.19.0
>
> Attachments: patch_multiple_workers_1.txt, patch_multiple_workers_2.txt, patch_multiple_workers_3.txt, patch_multiple_workers_4.txt
>
>
> Currently HOD launches at most one slave per node. For purposes of testing a large number of slaves on much fewer resources - for e.g. testing scalability of clusters, it will be useful if it can provision multiple slaves per node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3695) [HOD] Have an ability to run
multiple slaves per node
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612026#action_12612026 ]
Hadoop QA commented on HADOOP-3695:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12385603/patch_multiple_workers_4.txt
against trunk revision 675078.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2829/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2829/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2829/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2829/console
This message is automatically generated.
> [HOD] Have an ability to run multiple slaves per node
> -----------------------------------------------------
>
> Key: HADOOP-3695
> URL: https://issues.apache.org/jira/browse/HADOOP-3695
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.19.0
>
> Attachments: patch_multiple_workers_1.txt, patch_multiple_workers_2.txt, patch_multiple_workers_3.txt, patch_multiple_workers_4.txt
>
>
> Currently HOD launches at most one slave per node. For purposes of testing a large number of slaves on much fewer resources - for e.g. testing scalability of clusters, it will be useful if it can provision multiple slaves per node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3695) [HOD] Have an ability to run multiple
slaves per node
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-3695:
-------------------------------------
Status: Patch Available (was: Open)
> [HOD] Have an ability to run multiple slaves per node
> -----------------------------------------------------
>
> Key: HADOOP-3695
> URL: https://issues.apache.org/jira/browse/HADOOP-3695
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.19.0
>
> Attachments: patch_multiple_workers_1.txt, patch_multiple_workers_2.txt, patch_multiple_workers_3.txt, patch_multiple_workers_4.txt
>
>
> Currently HOD launches at most one slave per node. For purposes of testing a large number of slaves on much fewer resources - for e.g. testing scalability of clusters, it will be useful if it can provision multiple slaves per node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3695) [HOD] Have an ability to run multiple
slaves per node
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated HADOOP-3695:
--------------------------------------------
Attachment: patch_multiple_workers_3.txt
Attaching a new patch.
mapred.system.dir is used only for pulling job files and obtained by TTs from JT via RPC (from Hadoop 0.18 on. See HADOOP-3135 ). Any value in configuration files isn't used by TTs, and so it need not be different for multiple TTs running on the same host. In fact it doesn't need to be generated on TTs at all. Also, verified this with Devraj. We will have to generate a unique value for this too if we wish to support this feature for versions previous to 0.18.
Incorporated the rest of the three comments.
> [HOD] Have an ability to run multiple slaves per node
> -----------------------------------------------------
>
> Key: HADOOP-3695
> URL: https://issues.apache.org/jira/browse/HADOOP-3695
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.19.0
>
> Attachments: patch_multiple_workers_1.txt, patch_multiple_workers_2.txt, patch_multiple_workers_3.txt
>
>
> Currently HOD launches at most one slave per node. For purposes of testing a large number of slaves on much fewer resources - for e.g. testing scalability of clusters, it will be useful if it can provision multiple slaves per node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3695) [HOD] Have an ability to run
multiple slaves per node
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12624775#action_12624775 ]
Hudson commented on HADOOP-3695:
--------------------------------
Integrated in Hadoop-trunk #581 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/581/])
> [HOD] Have an ability to run multiple slaves per node
> -----------------------------------------------------
>
> Key: HADOOP-3695
> URL: https://issues.apache.org/jira/browse/HADOOP-3695
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.19.0
>
> Attachments: patch_multiple_workers_1.txt, patch_multiple_workers_2.txt, patch_multiple_workers_3.txt, patch_multiple_workers_4.txt
>
>
> Currently HOD launches at most one slave per node. For purposes of testing a large number of slaves on much fewer resources - for e.g. testing scalability of clusters, it will be useful if it can provision multiple slaves per node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3695) [HOD] Have an ability to run
multiple slaves per node
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12610722#action_12610722 ]
Hemanth Yamijala commented on HADOOP-3695:
------------------------------------------
Looks good. Minor points:
- In bin/hod, there is a check that tries to make sure ringmaster.workers_per_ring is atleast 1. But the check is incorrect, as it checks for an error condition with value <= 1, rather than < 1.
- Small formatting errors where workers_per_ring is defined bin/hod and bin/ringmaster files.
- While most of the local file system directories are generated with different names for the different workers, the mapred system directory is the same. We just need to make sure (with someone from the Map/Reduce team) that if tasktrackers from the same host use this system directory, there are no problems.
- logcondense.py needs to be enhanced. It now assumes that log file names will only have patterns like 0-datanode, 0-tasktracker or 1-tasktracker. In general, this can be [number]-datanode and [number]-tasktracker with this patch.
- We can enhance test cases in testRingmasterRPCs
- We should enhance documentation in the config guide.
> [HOD] Have an ability to run multiple slaves per node
> -----------------------------------------------------
>
> Key: HADOOP-3695
> URL: https://issues.apache.org/jira/browse/HADOOP-3695
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.19.0
>
> Attachments: patch_multiple_workers_1.txt
>
>
> Currently HOD launches at most one slave per node. For purposes of testing a large number of slaves on much fewer resources - for e.g. testing scalability of clusters, it will be useful if it can provision multiple slaves per node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3695) [HOD] Have an ability to run multiple
slaves per node
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-3695:
-------------------------------------
Attachment: patch_multiple_workers_2.txt
New patch that fixes the error check in bin/hod and indentation.
> [HOD] Have an ability to run multiple slaves per node
> -----------------------------------------------------
>
> Key: HADOOP-3695
> URL: https://issues.apache.org/jira/browse/HADOOP-3695
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.19.0
>
> Attachments: patch_multiple_workers_1.txt, patch_multiple_workers_2.txt
>
>
> Currently HOD launches at most one slave per node. For purposes of testing a large number of slaves on much fewer resources - for e.g. testing scalability of clusters, it will be useful if it can provision multiple slaves per node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3695) [HOD] Have an ability to run multiple
slaves per node
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated HADOOP-3695:
--------------------------------------------
Attachment: patch_multiple_workers_1.txt
Attaching patch for HOD to run multiple workers per hodring. The config parameter ringmaster.workers_per_ring specifies the number of workers (DNs and TTs) that each hodring should run. It defaults to 1.
Tested the patch on a 4 node cluster, with ringmaster.workers_per_ring set to 4 (total 2 masters, 8 workers per hadoop service) and it works fine.
> [HOD] Have an ability to run multiple slaves per node
> -----------------------------------------------------
>
> Key: HADOOP-3695
> URL: https://issues.apache.org/jira/browse/HADOOP-3695
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.19.0
>
> Attachments: patch_multiple_workers_1.txt
>
>
> Currently HOD launches at most one slave per node. For purposes of testing a large number of slaves on much fewer resources - for e.g. testing scalability of clusters, it will be useful if it can provision multiple slaves per node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3695) [HOD] Have an ability to run
multiple slaves per node
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12611946#action_12611946 ]
Hemanth Yamijala commented on HADOOP-3695:
------------------------------------------
+1 on the new patch, except for one small addition I would like to make to the documentation added for the configuration variable. Otherwise, it is good.
> [HOD] Have an ability to run multiple slaves per node
> -----------------------------------------------------
>
> Key: HADOOP-3695
> URL: https://issues.apache.org/jira/browse/HADOOP-3695
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.19.0
>
> Attachments: patch_multiple_workers_1.txt, patch_multiple_workers_2.txt, patch_multiple_workers_3.txt
>
>
> Currently HOD launches at most one slave per node. For purposes of testing a large number of slaves on much fewer resources - for e.g. testing scalability of clusters, it will be useful if it can provision multiple slaves per node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3695) [HOD] Have an ability to run multiple
slaves per node
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala updated HADOOP-3695:
-------------------------------------
Resolution: Fixed
Release Note: Added an ability in HOD to start multiple workers (TaskTrackers and/or DataNodes) per node to assist testing and simulation of scale. A configuration variable ringmaster.workers_per_ring was added to specify the number of workers to start.
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this to trunk. Thanks, Vinod !
> [HOD] Have an ability to run multiple slaves per node
> -----------------------------------------------------
>
> Key: HADOOP-3695
> URL: https://issues.apache.org/jira/browse/HADOOP-3695
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/hod
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.19.0
>
> Attachments: patch_multiple_workers_1.txt, patch_multiple_workers_2.txt, patch_multiple_workers_3.txt, patch_multiple_workers_4.txt
>
>
> Currently HOD launches at most one slave per node. For purposes of testing a large number of slaves on much fewer resources - for e.g. testing scalability of clusters, it will be useful if it can provision multiple slaves per node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.