You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Tom White (JIRA)" <ji...@apache.org> on 2009/12/22 23:48:29 UTC
[jira] Created: (HADOOP-6464) Write a Rackspace cloud provider
Write a Rackspace cloud provider
--------------------------------
Key: HADOOP-6464
URL: https://issues.apache.org/jira/browse/HADOOP-6464
Project: Hadoop Common
Issue Type: New Feature
Components: contrib/cloud
Reporter: Tom White
Assignee: Tom White
The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Rackspace Cloud Servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6464) Write a Rackspace cloud provider
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797980#action_12797980 ]
Hudson commented on HADOOP-6464:
--------------------------------
Integrated in Hadoop-Common-trunk #212 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/212/])
. Write a Rackspace cloud provider.
> Write a Rackspace cloud provider
> --------------------------------
>
> Key: HADOOP-6464
> URL: https://issues.apache.org/jira/browse/HADOOP-6464
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/cloud
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.22.0
>
> Attachments: HADOOP-6464.patch, HADOOP-6464.patch, HADOOP-6464.patch
>
>
> The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Rackspace Cloud Servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6464) Write a Rackspace cloud provider
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-6464:
------------------------------
Status: Patch Available (was: Open)
> Write a Rackspace cloud provider
> --------------------------------
>
> Key: HADOOP-6464
> URL: https://issues.apache.org/jira/browse/HADOOP-6464
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/cloud
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6464.patch, HADOOP-6464.patch
>
>
> The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Rackspace Cloud Servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6464) Write a Rackspace cloud provider
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-6464:
------------------------------
Attachment: HADOOP-6464.patch
This version of the patch should fix the release audit warnings. I've moved the Rackspace documentation into the top-level README (at some point the documentation should be broken out into a more structured format).
> Write a Rackspace cloud provider
> --------------------------------
>
> Key: HADOOP-6464
> URL: https://issues.apache.org/jira/browse/HADOOP-6464
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/cloud
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6464.patch, HADOOP-6464.patch, HADOOP-6464.patch
>
>
> The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Rackspace Cloud Servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6464) Write a Rackspace cloud provider
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-6464:
------------------------------
Attachment: HADOOP-6464.patch
This patch uses the Python libcloud API (http://incubator.apache.org/libcloud/) to communicate with Rackspace's API. Instrcutions are provided in the README.
> Write a Rackspace cloud provider
> --------------------------------
>
> Key: HADOOP-6464
> URL: https://issues.apache.org/jira/browse/HADOOP-6464
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/cloud
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6464.patch
>
>
> The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Rackspace Cloud Servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6464) Write a Rackspace cloud provider
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797273#action_12797273 ]
Hadoop QA commented on HADOOP-6464:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12429565/HADOOP-6464.patch
against trunk revision 896259.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/259/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/259/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/259/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/259/console
This message is automatically generated.
> Write a Rackspace cloud provider
> --------------------------------
>
> Key: HADOOP-6464
> URL: https://issues.apache.org/jira/browse/HADOOP-6464
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/cloud
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6464.patch, HADOOP-6464.patch, HADOOP-6464.patch
>
>
> The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Rackspace Cloud Servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6464) Write a Rackspace cloud provider
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796811#action_12796811 ]
Aaron Kimball commented on HADOOP-6464:
---------------------------------------
Can you please describe the test plan you used to actually test end-to-end coverage of rackspace?
regarding the patch:
see comments re. TERREMARK_KEY in HADOOP-6465 for RACKSPACE_KEY, etc.
re. boot script: I noticed a lot of very similar code in the terremark boot script. Can some of this be factored out into a common utility script that gets sourced in? This seems to be a looming maintenence problem.
RackspaceCluster.wait_for_instances() looks like a clone of TerremarkCluster.wait_for_instances(). Can this be pushed up into cluster?
Ditto with _all_started(), _ip_list_to_string(), terminate()?
Why is rackspace remote-setup.sh strictly-ubuntu-based? The boot script looks like it also handles centos.
> Write a Rackspace cloud provider
> --------------------------------
>
> Key: HADOOP-6464
> URL: https://issues.apache.org/jira/browse/HADOOP-6464
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/cloud
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6464.patch
>
>
> The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Rackspace Cloud Servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6464) Write a Rackspace cloud provider
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797234#action_12797234 ]
Hadoop QA commented on HADOOP-6464:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12429556/HADOOP-6464.patch
against trunk revision 896259.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 4 release audit warnings (more than the trunk's current 1 warnings).
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/258/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/258/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/258/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/258/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/258/console
This message is automatically generated.
> Write a Rackspace cloud provider
> --------------------------------
>
> Key: HADOOP-6464
> URL: https://issues.apache.org/jira/browse/HADOOP-6464
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/cloud
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6464.patch, HADOOP-6464.patch
>
>
> The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Rackspace Cloud Servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6464) Write a Rackspace cloud provider
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-6464:
------------------------------
Status: Open (was: Patch Available)
> Write a Rackspace cloud provider
> --------------------------------
>
> Key: HADOOP-6464
> URL: https://issues.apache.org/jira/browse/HADOOP-6464
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/cloud
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6464.patch, HADOOP-6464.patch, HADOOP-6464.patch
>
>
> The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Rackspace Cloud Servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6464) Write a Rackspace cloud provider
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-6464:
------------------------------
Attachment: HADOOP-6464.patch
> Can you please describe the test plan you used to actually test end-to-end coverage of rackspace?
I tested this by manually spinning up a cluster, running a job, checking the results and terminating the cluster. The commands to do this are in the README.
I've now modified the integration test script to run on Rackspace, which I successfully ran with the following (note that the image ID is private, since Rackspace does not support shared images):
{code}
HADOOP_CLOUD_PROVIDER=rackspace IMAGE_ID=200181 INSTANCE_TYPE=4 HADOOP_HOME=~/dev/hadoop-0.20.1/ ./transient-cluster.sh
{code}
> regarding the patch:
> see comments re. TERREMARK_KEY in HADOOP-6465 for RACKSPACE_KEY, etc.
Agree that this would be a useful way to specify keys. I've created HADOOP-6480 for this.
> re. boot script: I noticed a lot of very similar code in the terremark boot script. Can some of this be factored out into a common utility script that gets sourced in? This seems to be a looming maintenence problem.
This could become a problem, but I'd like to address it in a separate issue.
> RackspaceCluster.wait_for_instances() looks like a clone of TerremarkCluster.wait_for_instances(). Can this be pushed up into cluster?
> Ditto with _all_started(), _ip_list_to_string(), terminate()?
These are the same methods. I'd like to commit this one first, then I'll re-work HADOOP-6465 so the methods are pulled up into Cluster.
> Why is rackspace remote-setup.sh strictly-ubuntu-based? The boot script looks like it also handles centos.
I've only tested on Ubuntu so far, but it would be straightforward to run on Centos too. I'll create another JIRA for this.
Thanks for the review!
> Write a Rackspace cloud provider
> --------------------------------
>
> Key: HADOOP-6464
> URL: https://issues.apache.org/jira/browse/HADOOP-6464
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/cloud
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6464.patch, HADOOP-6464.patch
>
>
> The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Rackspace Cloud Servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6464) Write a Rackspace cloud provider
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797805#action_12797805 ]
Hudson commented on HADOOP-6464:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #138 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/138/])
. Write a Rackspace cloud provider.
> Write a Rackspace cloud provider
> --------------------------------
>
> Key: HADOOP-6464
> URL: https://issues.apache.org/jira/browse/HADOOP-6464
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/cloud
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.22.0
>
> Attachments: HADOOP-6464.patch, HADOOP-6464.patch, HADOOP-6464.patch
>
>
> The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Rackspace Cloud Servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6464) Write a Rackspace cloud provider
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-6464:
------------------------------
Resolution: Fixed
Fix Version/s: 0.22.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I've just committed this.
> Write a Rackspace cloud provider
> --------------------------------
>
> Key: HADOOP-6464
> URL: https://issues.apache.org/jira/browse/HADOOP-6464
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/cloud
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.22.0
>
> Attachments: HADOOP-6464.patch, HADOOP-6464.patch, HADOOP-6464.patch
>
>
> The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Rackspace Cloud Servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6464) Write a Rackspace cloud provider
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797751#action_12797751 ]
Aaron Kimball commented on HADOOP-6464:
---------------------------------------
Your intended refactorings sound reasonable. +1 on the patch
> Write a Rackspace cloud provider
> --------------------------------
>
> Key: HADOOP-6464
> URL: https://issues.apache.org/jira/browse/HADOOP-6464
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/cloud
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6464.patch, HADOOP-6464.patch, HADOOP-6464.patch
>
>
> The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Rackspace Cloud Servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6464) Write a Rackspace cloud provider
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-6464:
------------------------------
Status: Patch Available (was: Open)
> Write a Rackspace cloud provider
> --------------------------------
>
> Key: HADOOP-6464
> URL: https://issues.apache.org/jira/browse/HADOOP-6464
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/cloud
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6464.patch, HADOOP-6464.patch, HADOOP-6464.patch
>
>
> The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Rackspace Cloud Servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.