You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Tom White (JIRA)" <ji...@apache.org> on 2008/09/09 15:43:44 UTC
[jira] Created: (HADOOP-4131) Allow use of hadoop scripts on EC2
Allow use of hadoop scripts on EC2
----------------------------------
Key: HADOOP-4131
URL: https://issues.apache.org/jira/browse/HADOOP-4131
Project: Hadoop Core
Issue Type: Improvement
Components: contrib/ec2
Reporter: Tom White
Assignee: Tom White
Fix For: 0.19.0
Currently the Hadoop control scripts do not work on EC2 since instances do not have ssh access to one another. This is inconvenient as it makes it impossible to restart the cluster after a configuration change, for example.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4131) Allow use of hadoop scripts on EC2
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629489#action_12629489 ]
Tom White commented on HADOOP-4131:
-----------------------------------
A straightforward way of doing this would be to run Hadoop as a hadoop user, and export its home directory using NFS. The master node would be the NFS server, and the worker nodes would be NFS clients. This would ensure that the user's .ssh directory is shared, thus permitting ssh access between nodes.
> Allow use of hadoop scripts on EC2
> ----------------------------------
>
> Key: HADOOP-4131
> URL: https://issues.apache.org/jira/browse/HADOOP-4131
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/ec2
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.19.0
>
>
> Currently the Hadoop control scripts do not work on EC2 since instances do not have ssh access to one another. This is inconvenient as it makes it impossible to restart the cluster after a configuration change, for example.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4131) Allow use of hadoop scripts on EC2
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-4131:
------------------------------
Fix Version/s: (was: 0.19.0)
> Allow use of hadoop scripts on EC2
> ----------------------------------
>
> Key: HADOOP-4131
> URL: https://issues.apache.org/jira/browse/HADOOP-4131
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/ec2
> Reporter: Tom White
> Assignee: Tom White
>
> Currently the Hadoop control scripts do not work on EC2 since instances do not have ssh access to one another. This is inconvenient as it makes it impossible to restart the cluster after a configuration change, for example.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4131) Allow use of hadoop scripts on EC2
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-4131:
------------------------------
Attachment: hadoop-4131.patch
Here's a patch that does what the old scripts used to do: scps the private key to all machines so you can ssh between them, rsync config around, etc. It also generates a slaves file for the master node, so you can stop and start the cluster using the standard scripts.
> Allow use of hadoop scripts on EC2
> ----------------------------------
>
> Key: HADOOP-4131
> URL: https://issues.apache.org/jira/browse/HADOOP-4131
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/ec2
> Reporter: Tom White
> Assignee: Tom White
> Attachments: hadoop-4131.patch
>
>
> Currently the Hadoop control scripts do not work on EC2 since instances do not have ssh access to one another. This is inconvenient as it makes it impossible to restart the cluster after a configuration change, for example.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4131) Allow use of hadoop scripts on EC2
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682408#action_12682408 ]
Doug Cutting commented on HADOOP-4131:
--------------------------------------
A nit: 'update-slaves' sounds to me like it might make a substantial update to all the slave nodes. So maybe this would better be called 'update-slave-set' or somesuch?
> Allow use of hadoop scripts on EC2
> ----------------------------------
>
> Key: HADOOP-4131
> URL: https://issues.apache.org/jira/browse/HADOOP-4131
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/ec2
> Reporter: Tom White
> Assignee: Tom White
> Attachments: hadoop-4131-v2.patch, hadoop-4131.patch
>
>
> Currently the Hadoop control scripts do not work on EC2 since instances do not have ssh access to one another. This is inconvenient as it makes it impossible to restart the cluster after a configuration change, for example.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4131) Allow use of hadoop scripts on EC2
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-4131:
------------------------------
Attachment: hadoop-4131-v3.patch
Changed "update-slaves" to "update-slaves-file".
> Allow use of hadoop scripts on EC2
> ----------------------------------
>
> Key: HADOOP-4131
> URL: https://issues.apache.org/jira/browse/HADOOP-4131
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/ec2
> Reporter: Tom White
> Assignee: Tom White
> Attachments: hadoop-4131-v2.patch, hadoop-4131-v3.patch, hadoop-4131.patch
>
>
> Currently the Hadoop control scripts do not work on EC2 since instances do not have ssh access to one another. This is inconvenient as it makes it impossible to restart the cluster after a configuration change, for example.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4131) Allow use of hadoop scripts on EC2
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629512#action_12629512 ]
Steve Loughran commented on HADOOP-4131:
----------------------------------------
except now the node is exporting your ~/.ssh dir to anyone the firewall blocked hasn't blocked off who is willing to pretend to be you, which implies that you are giving full rights to anyone nearby.
Ideally, every client machine should pick up the public SSH key they trust from somewhere secure (metadata? /root/.ssh/authorized_keys); only the master(s) would have the private key needed to gain access.
> Allow use of hadoop scripts on EC2
> ----------------------------------
>
> Key: HADOOP-4131
> URL: https://issues.apache.org/jira/browse/HADOOP-4131
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/ec2
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.19.0
>
>
> Currently the Hadoop control scripts do not work on EC2 since instances do not have ssh access to one another. This is inconvenient as it makes it impossible to restart the cluster after a configuration change, for example.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4131) Allow use of hadoop scripts on EC2
Posted by "Philip Zeyliger (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682376#action_12682376 ]
Philip Zeyliger commented on HADOOP-4131:
-----------------------------------------
+1
> Allow use of hadoop scripts on EC2
> ----------------------------------
>
> Key: HADOOP-4131
> URL: https://issues.apache.org/jira/browse/HADOOP-4131
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/ec2
> Reporter: Tom White
> Assignee: Tom White
> Attachments: hadoop-4131-v2.patch, hadoop-4131.patch
>
>
> Currently the Hadoop control scripts do not work on EC2 since instances do not have ssh access to one another. This is inconvenient as it makes it impossible to restart the cluster after a configuration change, for example.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4131) Allow use of hadoop scripts on EC2
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-4131:
------------------------------
Attachment: hadoop-4131-v2.patch
Improved version that also updates a dfs.hosts file.
> Allow use of hadoop scripts on EC2
> ----------------------------------
>
> Key: HADOOP-4131
> URL: https://issues.apache.org/jira/browse/HADOOP-4131
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/ec2
> Reporter: Tom White
> Assignee: Tom White
> Attachments: hadoop-4131-v2.patch, hadoop-4131.patch
>
>
> Currently the Hadoop control scripts do not work on EC2 since instances do not have ssh access to one another. This is inconvenient as it makes it impossible to restart the cluster after a configuration change, for example.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.