You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@whirr.apache.org by "Tom White (JIRA)" <ji...@apache.org> on 2011/01/31 20:58:28 UTC
[jira] Created: (WHIRR-225) Support locally-supplied scripts
Support locally-supplied scripts
--------------------------------
Key: WHIRR-225
URL: https://issues.apache.org/jira/browse/WHIRR-225
Project: Whirr
Issue Type: New Feature
Reporter: Tom White
Assignee: Tom White
Whirr currently uses runurl to download bootstrap and configuration scripts from a webserver. It should be possible to use scripts supplied from the launch machine (or even from another source).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-225) Support locally-supplied scripts
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12991421#comment-12991421 ]
Tom White commented on WHIRR-225:
---------------------------------
> Would this also mean deprecating and eventually removing the runurl keys from each service?
Yes, the scripts would be re-written as functions and we would stop hosting them on S3. For larger clusters we would implement the blobstore optimization later.
> Support locally-supplied scripts
> --------------------------------
>
> Key: WHIRR-225
> URL: https://issues.apache.org/jira/browse/WHIRR-225
> Project: Whirr
> Issue Type: New Feature
> Reporter: Tom White
> Assignee: Tom White
> Attachments: WHIRR-225.patch
>
>
> Whirr currently uses runurl to download bootstrap and configuration scripts from a webserver. It should be possible to use scripts supplied from the launch machine (or even from another source).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (WHIRR-225) Support locally-supplied scripts
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated WHIRR-225:
----------------------------
Attachment: WHIRR-225.patch
First go at this. HBase, CDH don't yet work.
> Support locally-supplied scripts
> --------------------------------
>
> Key: WHIRR-225
> URL: https://issues.apache.org/jira/browse/WHIRR-225
> Project: Whirr
> Issue Type: New Feature
> Reporter: Tom White
> Assignee: Adrian Cole
> Fix For: 0.4.0
>
> Attachments: WHIRR-225.patch, WHIRR-225.patch
>
>
> Whirr currently uses runurl to download bootstrap and configuration scripts from a webserver. It should be possible to use scripts supplied from the launch machine (or even from another source).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-225) Support locally-supplied scripts
Posted by "Adrian Cole (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992364#comment-12992364 ]
Adrian Cole commented on WHIRR-225:
-----------------------------------
+1
tested on
aws-s3 cassandra,zookeeper,hadoop,hbase
cloudservers-us cassandra,zookeeper,hadoop
note hbase config doesn't seem setup for cloudservers, conceding this hasn't anything to do with this issue
> Support locally-supplied scripts
> --------------------------------
>
> Key: WHIRR-225
> URL: https://issues.apache.org/jira/browse/WHIRR-225
> Project: Whirr
> Issue Type: New Feature
> Reporter: Tom White
> Assignee: Adrian Cole
> Fix For: 0.4.0
>
> Attachments: WHIRR-225.patch, WHIRR-225.patch, WHIRR-225.patch
>
>
> Whirr currently uses runurl to download bootstrap and configuration scripts from a webserver. It should be possible to use scripts supplied from the launch machine (or even from another source).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (WHIRR-225) Support locally-supplied scripts
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated WHIRR-225:
----------------------------
Attachment: WHIRR-225.patch
We can implement this by using jclouds scriptbuilder classes. For example, by putting bash functions in files named after the function in _src/main/resources/functions_ we can call them with
{code}
addStatement(event, call("function-name"));
{code}
It's also makes it possible for users to override the function by editing the file in _functions_, e.g. _functions/function-name.sh_, (since the CLI puts the top-level directory on the front of the classpath, so any functions defined here will take precedence to the ones in the JAR).
If we refactor the existing scripts to be functions like this, then we will no longer have to serve them from a webserver. However, for larger clusters it may be preferable to download files from a webserver, so that the client doesn't have to send a potentially large script to each node in the cluster. Instead it could upload the script to a blobstore, then send a runurl command to each node in the cluster (see https://github.com/jclouds/jclouds/raw/master/demos/test/ComputeAndBlobStoreTogetherHappilyLiveTest.java for one way of doing this). We could do this enhancement as a separate JIRA.
I've attached a patch which shows how this would look for an example script function. It runs a (possibly user-defined) script in the bootstrap phase of ZooKeeper.
Thoughts?
> Support locally-supplied scripts
> --------------------------------
>
> Key: WHIRR-225
> URL: https://issues.apache.org/jira/browse/WHIRR-225
> Project: Whirr
> Issue Type: New Feature
> Reporter: Tom White
> Assignee: Tom White
> Attachments: WHIRR-225.patch
>
>
> Whirr currently uses runurl to download bootstrap and configuration scripts from a webserver. It should be possible to use scripts supplied from the launch machine (or even from another source).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (WHIRR-225) Support locally-supplied scripts
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White resolved WHIRR-225.
-----------------------------
Resolution: Fixed
Assignee: Tom White (was: Adrian Cole)
I've just committed this.
> Support locally-supplied scripts
> --------------------------------
>
> Key: WHIRR-225
> URL: https://issues.apache.org/jira/browse/WHIRR-225
> Project: Whirr
> Issue Type: New Feature
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.4.0
>
> Attachments: WHIRR-225.patch, WHIRR-225.patch, WHIRR-225.patch, WHIRR-225.patch
>
>
> Whirr currently uses runurl to download bootstrap and configuration scripts from a webserver. It should be possible to use scripts supplied from the launch machine (or even from another source).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-225) Support locally-supplied scripts
Posted by "Adrian Cole (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12991277#comment-12991277 ]
Adrian Cole commented on WHIRR-225:
-----------------------------------
I like this approach. BlobStore functionality is an option we can use under the scenes as necessary. In other words, I'd start with doing everything in Statements and then optimize with BlobStore under the scenes as necessary.
> Support locally-supplied scripts
> --------------------------------
>
> Key: WHIRR-225
> URL: https://issues.apache.org/jira/browse/WHIRR-225
> Project: Whirr
> Issue Type: New Feature
> Reporter: Tom White
> Assignee: Tom White
> Attachments: WHIRR-225.patch
>
>
> Whirr currently uses runurl to download bootstrap and configuration scripts from a webserver. It should be possible to use scripts supplied from the launch machine (or even from another source).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-225) Support locally-supplied scripts
Posted by "Adrian Cole (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992383#comment-12992383 ]
Adrian Cole commented on WHIRR-225:
-----------------------------------
I meant that the test properties file has parameters that are only relevant to ec2. Ex. locationid. this shouldn't be specified in rackspace and overriding it to nothing is a bit awkward. thanks for opening WHIRR-233 on this. Again, this isn't anything new or specific to WHIRR-225
> Support locally-supplied scripts
> --------------------------------
>
> Key: WHIRR-225
> URL: https://issues.apache.org/jira/browse/WHIRR-225
> Project: Whirr
> Issue Type: New Feature
> Reporter: Tom White
> Assignee: Adrian Cole
> Fix For: 0.4.0
>
> Attachments: WHIRR-225.patch, WHIRR-225.patch, WHIRR-225.patch
>
>
> Whirr currently uses runurl to download bootstrap and configuration scripts from a webserver. It should be possible to use scripts supplied from the launch machine (or even from another source).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-225) Support locally-supplied scripts
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992384#comment-12992384 ]
Andrei Savu commented on WHIRR-225:
-----------------------------------
Looks great. I believe that in {{core/src/main/resources/functions/install_runurl.sh}} we should rename {{installRunUrl}} to {{install_runurl}}. Why are we keeping this function around?
> Support locally-supplied scripts
> --------------------------------
>
> Key: WHIRR-225
> URL: https://issues.apache.org/jira/browse/WHIRR-225
> Project: Whirr
> Issue Type: New Feature
> Reporter: Tom White
> Assignee: Adrian Cole
> Fix For: 0.4.0
>
> Attachments: WHIRR-225.patch, WHIRR-225.patch, WHIRR-225.patch
>
>
> Whirr currently uses runurl to download bootstrap and configuration scripts from a webserver. It should be possible to use scripts supplied from the launch machine (or even from another source).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-225) Support locally-supplied scripts
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992369#comment-12992369 ]
Lars George commented on WHIRR-225:
-----------------------------------
+1
Thanks Tom for doing this.
@Adrian:
bq. note hbase config doesn't seem setup for cloudservers, conceding this hasn't anything to do with this issue
What do you mean?
> Support locally-supplied scripts
> --------------------------------
>
> Key: WHIRR-225
> URL: https://issues.apache.org/jira/browse/WHIRR-225
> Project: Whirr
> Issue Type: New Feature
> Reporter: Tom White
> Assignee: Adrian Cole
> Fix For: 0.4.0
>
> Attachments: WHIRR-225.patch, WHIRR-225.patch, WHIRR-225.patch
>
>
> Whirr currently uses runurl to download bootstrap and configuration scripts from a webserver. It should be possible to use scripts supplied from the launch machine (or even from another source).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Assigned: (WHIRR-225) Support locally-supplied scripts
Posted by "Adrian Cole (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adrian Cole reassigned WHIRR-225:
---------------------------------
Assignee: Adrian Cole (was: Tom White)
> Support locally-supplied scripts
> --------------------------------
>
> Key: WHIRR-225
> URL: https://issues.apache.org/jira/browse/WHIRR-225
> Project: Whirr
> Issue Type: New Feature
> Reporter: Tom White
> Assignee: Adrian Cole
> Attachments: WHIRR-225.patch
>
>
> Whirr currently uses runurl to download bootstrap and configuration scripts from a webserver. It should be possible to use scripts supplied from the launch machine (or even from another source).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (WHIRR-225) Support locally-supplied scripts
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated WHIRR-225:
----------------------------
Fix Version/s: 0.4.0
> Support locally-supplied scripts
> --------------------------------
>
> Key: WHIRR-225
> URL: https://issues.apache.org/jira/browse/WHIRR-225
> Project: Whirr
> Issue Type: New Feature
> Reporter: Tom White
> Assignee: Adrian Cole
> Fix For: 0.4.0
>
> Attachments: WHIRR-225.patch
>
>
> Whirr currently uses runurl to download bootstrap and configuration scripts from a webserver. It should be possible to use scripts supplied from the launch machine (or even from another source).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-225) Support locally-supplied scripts
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12991368#comment-12991368 ]
Lars George commented on WHIRR-225:
-----------------------------------
Would this also mean deprecating and eventually removing the runurl keys from each service? I see no reason to keep them to be honest and we would need to split the code into two, one supporting runurl the other the scriptbuilder API.
> Support locally-supplied scripts
> --------------------------------
>
> Key: WHIRR-225
> URL: https://issues.apache.org/jira/browse/WHIRR-225
> Project: Whirr
> Issue Type: New Feature
> Reporter: Tom White
> Assignee: Tom White
> Attachments: WHIRR-225.patch
>
>
> Whirr currently uses runurl to download bootstrap and configuration scripts from a webserver. It should be possible to use scripts supplied from the launch machine (or even from another source).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (WHIRR-225) Support locally-supplied scripts
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated WHIRR-225:
----------------------------
Attachment: WHIRR-225.patch
This version passes integration tests. It doesn't remove the scripts directory, but we should do this as a part of this issue. There are some Cassandra scripts in scripts/apache/cassandra (nodetool, start, stop, wipe-state) that are not used - do we need to keep them, or should we delete them until we have some way of running them on the cluster?
> Support locally-supplied scripts
> --------------------------------
>
> Key: WHIRR-225
> URL: https://issues.apache.org/jira/browse/WHIRR-225
> Project: Whirr
> Issue Type: New Feature
> Reporter: Tom White
> Assignee: Adrian Cole
> Fix For: 0.4.0
>
> Attachments: WHIRR-225.patch, WHIRR-225.patch, WHIRR-225.patch
>
>
> Whirr currently uses runurl to download bootstrap and configuration scripts from a webserver. It should be possible to use scripts supplied from the launch machine (or even from another source).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (WHIRR-225) Support locally-supplied scripts
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated WHIRR-225:
----------------------------
Attachment: WHIRR-225.patch
Minor update that removes the scripts (except for the extra Cassandra ones), and reinstates a test teardown I had removed during testing.
> Support locally-supplied scripts
> --------------------------------
>
> Key: WHIRR-225
> URL: https://issues.apache.org/jira/browse/WHIRR-225
> Project: Whirr
> Issue Type: New Feature
> Reporter: Tom White
> Assignee: Adrian Cole
> Fix For: 0.4.0
>
> Attachments: WHIRR-225.patch, WHIRR-225.patch, WHIRR-225.patch, WHIRR-225.patch
>
>
> Whirr currently uses runurl to download bootstrap and configuration scripts from a webserver. It should be possible to use scripts supplied from the launch machine (or even from another source).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (WHIRR-225) Support locally-supplied scripts
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992512#comment-12992512 ]
Tom White commented on WHIRR-225:
---------------------------------
> we should rename installRunUrl to install_runurl.
The patch should do this (I'll make sure it happens at commit time).
> Why are we keeping this function around?
We should still allow scripts to use runurl if they wish, so I think we should keep it.
I'll commit this later today. Thanks for the reviews everyone.
> Support locally-supplied scripts
> --------------------------------
>
> Key: WHIRR-225
> URL: https://issues.apache.org/jira/browse/WHIRR-225
> Project: Whirr
> Issue Type: New Feature
> Reporter: Tom White
> Assignee: Adrian Cole
> Fix For: 0.4.0
>
> Attachments: WHIRR-225.patch, WHIRR-225.patch, WHIRR-225.patch
>
>
> Whirr currently uses runurl to download bootstrap and configuration scripts from a webserver. It should be possible to use scripts supplied from the launch machine (or even from another source).
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira