You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Tom White (JIRA)" <ji...@apache.org> on 2009/12/22 23:49:30 UTC
[jira] Created: (HADOOP-6465) Write a Terremark cloud provider
Write a Terremark cloud provider
--------------------------------
Key: HADOOP-6465
URL: https://issues.apache.org/jira/browse/HADOOP-6465
Project: Hadoop Common
Issue Type: New Feature
Components: contrib/cloud
Reporter: Tom White
Assignee: Tom White
The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Terremark's vCloud Express platform.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6465) Write a Terremark cloud provider
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-6465:
------------------------------
Status: Open (was: Patch Available)
> Write a Terremark cloud provider
> --------------------------------
>
> Key: HADOOP-6465
> URL: https://issues.apache.org/jira/browse/HADOOP-6465
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/cloud
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6465.patch, HADOOP-6465.patch, HADOOP-6465.patch
>
>
> The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Terremark's vCloud Express platform.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6465) Write a Terremark cloud provider
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12828964#action_12828964 ]
Hadoop QA commented on HADOOP-6465:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12430294/HADOOP-6465.patch
against trunk revision 905860.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 2 release audit warnings (more than the trunk's current 1 warnings).
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/321/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/321/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/321/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/321/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/321/console
This message is automatically generated.
> Write a Terremark cloud provider
> --------------------------------
>
> Key: HADOOP-6465
> URL: https://issues.apache.org/jira/browse/HADOOP-6465
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/cloud
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6465.patch, HADOOP-6465.patch
>
>
> The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Terremark's vCloud Express platform.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6465) Write a Terremark cloud provider
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-6465:
------------------------------
Attachment: HADOOP-6465.patch
Here's a new patch that addresses most of Aaron's feedback (see also HADOOP-6464 for responses to particular feedback items). This change depends on LIBCLOUD-5.
> Write a Terremark cloud provider
> --------------------------------
>
> Key: HADOOP-6465
> URL: https://issues.apache.org/jira/browse/HADOOP-6465
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/cloud
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6465.patch, HADOOP-6465.patch
>
>
> The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Terremark's vCloud Express platform.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6465) Write a Terremark cloud provider
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-6465:
------------------------------
Hadoop Flags: [Reviewed]
Status: Patch Available (was: Open)
> Write a Terremark cloud provider
> --------------------------------
>
> Key: HADOOP-6465
> URL: https://issues.apache.org/jira/browse/HADOOP-6465
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/cloud
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6465.patch, HADOOP-6465.patch
>
>
> The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Terremark's vCloud Express platform.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6465) Write a Terremark cloud provider
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-6465:
------------------------------
Attachment: HADOOP-6465.patch
New patch to fix the warning. I also made a minor change to the bootstrap script since the previous version no longer worked with Terremark.
> Write a Terremark cloud provider
> --------------------------------
>
> Key: HADOOP-6465
> URL: https://issues.apache.org/jira/browse/HADOOP-6465
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/cloud
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6465.patch, HADOOP-6465.patch, HADOOP-6465.patch
>
>
> The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Terremark's vCloud Express platform.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6465) Write a Terremark cloud provider
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-6465:
----------------------------------
Status: Open (was: Patch Available)
Please fix the release audit warning
> Write a Terremark cloud provider
> --------------------------------
>
> Key: HADOOP-6465
> URL: https://issues.apache.org/jira/browse/HADOOP-6465
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/cloud
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6465.patch, HADOOP-6465.patch
>
>
> The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Terremark's vCloud Express platform.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6465) Write a Terremark cloud provider
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-6465:
------------------------------
Status: Patch Available (was: Open)
> Write a Terremark cloud provider
> --------------------------------
>
> Key: HADOOP-6465
> URL: https://issues.apache.org/jira/browse/HADOOP-6465
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/cloud
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6465.patch, HADOOP-6465.patch, HADOOP-6465.patch
>
>
> The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Terremark's vCloud Express platform.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6465) Write a Terremark cloud provider
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796806#action_12796806 ]
Aaron Kimball commented on HADOOP-6465:
---------------------------------------
Tom,
Can you talk a bit about how you tested this? I see some unit tests, but that doesn't actually cover interaction with a terremark service.
As for the patch:
readme-vcloud:
regarding {{TERREMARK_KEY/SECRET}}, are these also specifiable via command-line options? I feel like they should be. (Ditto with AWS credentials if they aren't already.)
.. This should also be specifiable in the configuration file.
{{ssh-keygen -f id_rsa_rackspace -P ''}} -- rackspace? Shouldn't this example say "terremark"?
{quote}
Note: you should use short cluster name identifiers, here "tm", (no more than
four characters), since they are used as a part of the nstance name, which
is limited to 15 characters in Terremark.
{quote}
The program itself should warn you about this, if it doesn't already. (If this is going to cause users problems, don't count on them having read the README.)
hadoop-terremark-init-remote.sh:
In update_repo(), you have {{sudo apt-get update}} and {{yum update -y yum}}. If one requires the {{sudo}}, so should the other. Also, shouldn't they both take a {{-y}} argument?
Function {{install_java()}} -- this looks like it only works with {{dpkg}}. Is there a {{yum}}-based equivalent? If one is not necessary, put that in a comment?
install_base_packages() similarly is debian-specific and does not have a yum equivalent?
make_hadoop_dirs() allows multiple hadoop mount points. configure_hadoop is hard-coded to {{/data}} -- shouldn't this be parameterized too?
"Hadoop logs should be on the /mnt partition" -- you mean {{/data}} ?
start_namenode() and start_daemon() both include redundant logic to determine the value of {{$AS_HADOOP}} - consider factoring into a method.
vcloud.py:
you perform re.match() statements to dissect node-names of the form cluster-role-nodeId. Can you use some constants so instead of {{re.group(1)}} you have {{re.group(ROLE_PART)}}, CLUSTER_PART, NODE_ID_PART, etc?
> Write a Terremark cloud provider
> --------------------------------
>
> Key: HADOOP-6465
> URL: https://issues.apache.org/jira/browse/HADOOP-6465
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/cloud
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6465.patch
>
>
> The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Terremark's vCloud Express platform.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6465) Write a Terremark cloud provider
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836195#action_12836195 ]
Hadoop QA commented on HADOOP-6465:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12436436/HADOOP-6465.patch
against trunk revision 912056.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/20/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/20/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/20/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/20/console
This message is automatically generated.
> Write a Terremark cloud provider
> --------------------------------
>
> Key: HADOOP-6465
> URL: https://issues.apache.org/jira/browse/HADOOP-6465
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/cloud
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6465.patch, HADOOP-6465.patch, HADOOP-6465.patch
>
>
> The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Terremark's vCloud Express platform.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6465) Write a Terremark cloud provider
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-6465:
------------------------------
Attachment: HADOOP-6465.patch
This patch uses libcloud to communicate with the Terremark vCloud API.
> Write a Terremark cloud provider
> --------------------------------
>
> Key: HADOOP-6465
> URL: https://issues.apache.org/jira/browse/HADOOP-6465
> Project: Hadoop Common
> Issue Type: New Feature
> Components: contrib/cloud
> Reporter: Tom White
> Assignee: Tom White
> Attachments: HADOOP-6465.patch
>
>
> The scripts in contrib/cloud currently only support running on EC2. This issue is to add support for running Hadoop clusters on Terremark's vCloud Express platform.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.