You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Tom White (JIRA)" <ji...@apache.org> on 2009/11/25 21:50:39 UTC
[jira] Created: (HADOOP-6392) Run namenode and jobtracker on
separate EC2 instances
Run namenode and jobtracker on separate EC2 instances
-----------------------------------------------------
Key: HADOOP-6392
URL: https://issues.apache.org/jira/browse/HADOOP-6392
Project: Hadoop Common
Issue Type: New Feature
Components: contrib/ec2
Reporter: Tom White
Assignee: Tom White
Replace concept of "master" with that of "namenode" and "jobtracker". Still need to be able to run both on one node, of course.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6392) Run namenode and jobtracker on
separate EC2 instances
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787870#action_12787870 ]
Hadoop QA commented on HADOOP-6392:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12427398/HADOOP-6392.patch
against trunk revision 888565.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/179/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/179/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/179/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/179/console
This message is automatically generated.
> Run namenode and jobtracker on separate EC2 instances
> -----------------------------------------------------
>
> Key: HADOOP-6392
> URL: https://issues.apache.org/jira/browse/HADOOP-6392
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/ec2
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6392.patch
>
>
> Replace concept of "master" with that of "namenode" and "jobtracker". Still need to be able to run both on one node, of course.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6392) Run namenode and jobtracker on
separate EC2 instances
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782645#action_12782645 ]
Tom White commented on HADOOP-6392:
-----------------------------------
Colocation is the default, but splitting the namenode/jobtracker is better for large clusters. Good point about using a larger instance for the master - I've opened HADOOP-6393 for this.
> Run namenode and jobtracker on separate EC2 instances
> -----------------------------------------------------
>
> Key: HADOOP-6392
> URL: https://issues.apache.org/jira/browse/HADOOP-6392
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/ec2
> Reporter: Tom White
> Assignee: Tom White
>
> Replace concept of "master" with that of "namenode" and "jobtracker". Still need to be able to run both on one node, of course.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6392) Run namenode and jobtracker on
separate EC2 instances
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-6392:
------------------------------
Status: Patch Available (was: Open)
> Run namenode and jobtracker on separate EC2 instances
> -----------------------------------------------------
>
> Key: HADOOP-6392
> URL: https://issues.apache.org/jira/browse/HADOOP-6392
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/ec2
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6392.patch
>
>
> Replace concept of "master" with that of "namenode" and "jobtracker". Still need to be able to run both on one node, of course.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6392) Run namenode and jobtracker on
separate EC2 instances
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787903#action_12787903 ]
Tom White commented on HADOOP-6392:
-----------------------------------
I'll commit this tomorrow unless there are any objections.
> Run namenode and jobtracker on separate EC2 instances
> -----------------------------------------------------
>
> Key: HADOOP-6392
> URL: https://issues.apache.org/jira/browse/HADOOP-6392
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/ec2
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6392.patch
>
>
> Replace concept of "master" with that of "namenode" and "jobtracker". Still need to be able to run both on one node, of course.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6392) Run namenode and jobtracker on
separate EC2 instances
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-6392:
------------------------------
Attachment: HADOOP-6392.patch
Here's a patch that removes the strong requirement to have "master" and "slave" types. Instead, instances have one or more roles, which are stored in EC2 as security groups. The instance uses knowledge of its role to start the relevant daemons at boot time - for example the "nn" role will start up the namenode.
Here's how you launch one master node (running a namenode, secondary namenode, and a jobtracker) and 10 datanodes/tasktrackers:
{code}
hadoop-ec2 launch-cluster my-hadoop-cluster 1 nn,snn,jt 10 dn,tt
{code}
The following command will launch a namenode and a separate jobtracker (and 10 datanodes/tasktrackers).
{code}
hadoop-ec2 launch-cluster my-hadoop-cluster 1 nn,snn 1 jt 10 dn,tt
{code}
This patch is fairly large, but there is little new code, since it is mainly refactoring. Previously, the logic for each command was spread between cli.py and commands.py. Now, cli.py is responsible only for command line parsing, and it delegates to service.py, which contains the class HadoopService.py for the Hadoop-specific logic.
> Run namenode and jobtracker on separate EC2 instances
> -----------------------------------------------------
>
> Key: HADOOP-6392
> URL: https://issues.apache.org/jira/browse/HADOOP-6392
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/ec2
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6392.patch
>
>
> Replace concept of "master" with that of "namenode" and "jobtracker". Still need to be able to run both on one node, of course.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6392) Run namenode and jobtracker on
separate EC2 instances
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12782619#action_12782619 ]
Matei Zaharia commented on HADOOP-6392:
---------------------------------------
Just curious, what's the reasoning for this? I would imagine that the NN communicates with the JT a lot and thus it helps to colocate them. Another useful option might be to allow the user to ask for a bigger instance type for the "master" node (if the script doesn't currently allow this).
> Run namenode and jobtracker on separate EC2 instances
> -----------------------------------------------------
>
> Key: HADOOP-6392
> URL: https://issues.apache.org/jira/browse/HADOOP-6392
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/ec2
> Reporter: Tom White
> Assignee: Tom White
>
> Replace concept of "master" with that of "namenode" and "jobtracker". Still need to be able to run both on one node, of course.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6392) Run namenode and jobtracker on
separate EC2 instances
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788344#action_12788344 ]
Hudson commented on HADOOP-6392:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #102 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/102/])
. Run namenode and jobtracker on separate EC2 instances.
> Run namenode and jobtracker on separate EC2 instances
> -----------------------------------------------------
>
> Key: HADOOP-6392
> URL: https://issues.apache.org/jira/browse/HADOOP-6392
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/ec2
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.22.0
>
> Attachments: HADOOP-6392.patch
>
>
> Replace concept of "master" with that of "namenode" and "jobtracker". Still need to be able to run both on one node, of course.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6392) Run namenode and jobtracker on
separate EC2 instances
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788336#action_12788336 ]
Tom White commented on HADOOP-6392:
-----------------------------------
I've just committed this.
> Run namenode and jobtracker on separate EC2 instances
> -----------------------------------------------------
>
> Key: HADOOP-6392
> URL: https://issues.apache.org/jira/browse/HADOOP-6392
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/ec2
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.22.0
>
> Attachments: HADOOP-6392.patch
>
>
> Replace concept of "master" with that of "namenode" and "jobtracker". Still need to be able to run both on one node, of course.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6392) Run namenode and jobtracker on
separate EC2 instances
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-6392:
------------------------------
Resolution: Fixed
Fix Version/s: 0.22.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
> Run namenode and jobtracker on separate EC2 instances
> -----------------------------------------------------
>
> Key: HADOOP-6392
> URL: https://issues.apache.org/jira/browse/HADOOP-6392
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/ec2
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.22.0
>
> Attachments: HADOOP-6392.patch
>
>
> Replace concept of "master" with that of "namenode" and "jobtracker". Still need to be able to run both on one node, of course.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6392) Run namenode and jobtracker on
separate EC2 instances
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788680#action_12788680 ]
Hudson commented on HADOOP-6392:
--------------------------------
Integrated in Hadoop-Common-trunk #183 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/183/])
. Run namenode and jobtracker on separate EC2 instances.
> Run namenode and jobtracker on separate EC2 instances
> -----------------------------------------------------
>
> Key: HADOOP-6392
> URL: https://issues.apache.org/jira/browse/HADOOP-6392
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/ec2
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.22.0
>
> Attachments: HADOOP-6392.patch
>
>
> Replace concept of "master" with that of "namenode" and "jobtracker". Still need to be able to run both on one node, of course.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6392) Run namenode and jobtracker on
separate EC2 instances
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12787860#action_12787860 ]
Aaron Kimball commented on HADOOP-6392:
---------------------------------------
+1 looks good to me.
> Run namenode and jobtracker on separate EC2 instances
> -----------------------------------------------------
>
> Key: HADOOP-6392
> URL: https://issues.apache.org/jira/browse/HADOOP-6392
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/ec2
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6392.patch
>
>
> Replace concept of "master" with that of "namenode" and "jobtracker". Still need to be able to run both on one node, of course.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.