You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Andrew Klochkov (JIRA)" <ji...@apache.org> on 2010/04/09 13:49:51 UTC
[jira] Created: (HADOOP-6696) Use contrib/cloud scripts for
hardware cluster deployment
Use contrib/cloud scripts for hardware cluster deployment
---------------------------------------------------------
Key: HADOOP-6696
URL: https://issues.apache.org/jira/browse/HADOOP-6696
Project: Hadoop Common
Issue Type: Improvement
Components: contrib/cloud
Reporter: Andrew Klochkov
Assignee: Andrew Klochkov
contrib/cloud scripts can be used as an easy way to deploy Hadoop on a hardware cluster, by implementing a "fixed" provider. Here "fixed" means it uses a preconfigured set of hosts to deploy to.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6696) Use contrib/cloud scripts for
hardware cluster deployment
Posted by "Andrew Klochkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Klochkov updated HADOOP-6696:
------------------------------------
Status: Patch Available (was: Open)
> Use contrib/cloud scripts for hardware cluster deployment
> ---------------------------------------------------------
>
> Key: HADOOP-6696
> URL: https://issues.apache.org/jira/browse/HADOOP-6696
> Project: Hadoop Common
> Issue Type: Improvement
> Components: contrib/cloud
> Reporter: Andrew Klochkov
> Assignee: Andrew Klochkov
> Attachments: HADOOP-6696.patch, HADOOP-6696.patch
>
>
> contrib/cloud scripts can be used as an easy way to deploy Hadoop on a hardware cluster, by implementing a "fixed" provider. Here "fixed" means it uses a preconfigured set of hosts to deploy to.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6696) Use contrib/cloud scripts for
hardware cluster deployment
Posted by "Andrew Klochkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Klochkov updated HADOOP-6696:
------------------------------------
Attachment: HADOOP-6696.patch
Attached a patch.
Some notes on this implementation:
1. It enables Hadoop service deployment only, it doesn't work for zookeeper.
2. Configuration is done in $HOME/.hadoop-cloud/clusters.cfg.
Sample config is below. It's set to run jobtracker, namenode and secondary namenode on 172.16.128.203, and run 2 slaves on 172.16.128.206 and 172.16.128.239. Parameter NUM_SLAVES of the launch-cluster command becomes meaningless in case of fixed provider. README is not modified.
[fixed-cluster]
cloud_provider=fixed
image_id=
instance_type=
key_name=
availability_zone=
public_key=/keys/location/hadoop.pem
private_key=/keys/location/hadoop.pem
ssh_options=-i %(private_key)s -o StrictHostKeyChecking=no
nodes.nn_snn_jt=172.16.128.203
nodes.dn_tt=172.16.128.206, 172.16.128.239
env=MAX_MAP_TASKS=2
env=MAX_REDUCE_TASKS=1
3. Files hadoop-ec2-init-remote.sh and hadoop-fixed-init-remote.sh in "data" dir are almost identical. Obviously it makes sense to extract common code in a separate scripts, and later use it by other providers too.
> Use contrib/cloud scripts for hardware cluster deployment
> ---------------------------------------------------------
>
> Key: HADOOP-6696
> URL: https://issues.apache.org/jira/browse/HADOOP-6696
> Project: Hadoop Common
> Issue Type: Improvement
> Components: contrib/cloud
> Reporter: Andrew Klochkov
> Assignee: Andrew Klochkov
> Attachments: HADOOP-6696.patch
>
>
> contrib/cloud scripts can be used as an easy way to deploy Hadoop on a hardware cluster, by implementing a "fixed" provider. Here "fixed" means it uses a preconfigured set of hosts to deploy to.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6696) Use contrib/cloud scripts for
hardware cluster deployment
Posted by "Andrew Klochkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Klochkov updated HADOOP-6696:
------------------------------------
Status: Patch Available (was: Open)
> Use contrib/cloud scripts for hardware cluster deployment
> ---------------------------------------------------------
>
> Key: HADOOP-6696
> URL: https://issues.apache.org/jira/browse/HADOOP-6696
> Project: Hadoop Common
> Issue Type: Improvement
> Components: contrib/cloud
> Reporter: Andrew Klochkov
> Assignee: Andrew Klochkov
> Attachments: HADOOP-6696.patch
>
>
> contrib/cloud scripts can be used as an easy way to deploy Hadoop on a hardware cluster, by implementing a "fixed" provider. Here "fixed" means it uses a preconfigured set of hosts to deploy to.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6696) Use contrib/cloud scripts for
hardware cluster deployment
Posted by "Andrew Klochkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Klochkov updated HADOOP-6696:
------------------------------------
Attachment: HADOOP-6696.patch
First patch was incorrect. Attaching a correct one.
> Use contrib/cloud scripts for hardware cluster deployment
> ---------------------------------------------------------
>
> Key: HADOOP-6696
> URL: https://issues.apache.org/jira/browse/HADOOP-6696
> Project: Hadoop Common
> Issue Type: Improvement
> Components: contrib/cloud
> Reporter: Andrew Klochkov
> Assignee: Andrew Klochkov
> Attachments: HADOOP-6696.patch, HADOOP-6696.patch
>
>
> contrib/cloud scripts can be used as an easy way to deploy Hadoop on a hardware cluster, by implementing a "fixed" provider. Here "fixed" means it uses a preconfigured set of hosts to deploy to.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6696) Use contrib/cloud scripts for
hardware cluster deployment
Posted by "Andrew Klochkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Klochkov updated HADOOP-6696:
------------------------------------
Status: Open (was: Patch Available)
> Use contrib/cloud scripts for hardware cluster deployment
> ---------------------------------------------------------
>
> Key: HADOOP-6696
> URL: https://issues.apache.org/jira/browse/HADOOP-6696
> Project: Hadoop Common
> Issue Type: Improvement
> Components: contrib/cloud
> Reporter: Andrew Klochkov
> Assignee: Andrew Klochkov
> Attachments: HADOOP-6696.patch
>
>
> contrib/cloud scripts can be used as an easy way to deploy Hadoop on a hardware cluster, by implementing a "fixed" provider. Here "fixed" means it uses a preconfigured set of hosts to deploy to.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6696) Use contrib/cloud scripts for
hardware cluster deployment
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855365#action_12855365 ]
Hadoop QA commented on HADOOP-6696:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12441291/HADOOP-6696.patch
against trunk revision 932115.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/453/console
This message is automatically generated.
> Use contrib/cloud scripts for hardware cluster deployment
> ---------------------------------------------------------
>
> Key: HADOOP-6696
> URL: https://issues.apache.org/jira/browse/HADOOP-6696
> Project: Hadoop Common
> Issue Type: Improvement
> Components: contrib/cloud
> Reporter: Andrew Klochkov
> Assignee: Andrew Klochkov
> Attachments: HADOOP-6696.patch
>
>
> contrib/cloud scripts can be used as an easy way to deploy Hadoop on a hardware cluster, by implementing a "fixed" provider. Here "fixed" means it uses a preconfigured set of hosts to deploy to.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6696) Use contrib/cloud scripts for
hardware cluster deployment
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855383#action_12855383 ]
Hadoop QA commented on HADOOP-6696:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12441292/HADOOP-6696.patch
against trunk revision 932115.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/454/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/454/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/454/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/454/console
This message is automatically generated.
> Use contrib/cloud scripts for hardware cluster deployment
> ---------------------------------------------------------
>
> Key: HADOOP-6696
> URL: https://issues.apache.org/jira/browse/HADOOP-6696
> Project: Hadoop Common
> Issue Type: Improvement
> Components: contrib/cloud
> Reporter: Andrew Klochkov
> Assignee: Andrew Klochkov
> Attachments: HADOOP-6696.patch, HADOOP-6696.patch
>
>
> contrib/cloud scripts can be used as an easy way to deploy Hadoop on a hardware cluster, by implementing a "fixed" provider. Here "fixed" means it uses a preconfigured set of hosts to deploy to.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.