You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@whirr.apache.org by "Andrei Savu (JIRA)" <ji...@apache.org> on 2011/04/28 16:35:03 UTC
[jira] [Created] (WHIRR-288) Add blob store persistence for cluster
state
Add blob store persistence for cluster state
--------------------------------------------
Key: WHIRR-288
URL: https://issues.apache.org/jira/browse/WHIRR-288
Project: Whirr
Issue Type: Improvement
Reporter: Andrei Savu
Assignee: Andrei Savu
Fix For: 0.5.0
Right now Whirr persists the current cluster state (instance IDs, public and private IPs and roles) on the local filesystem (~/.whirr/<cluster-name>/instances). It should be possible to save the state as a blob so that it's easily accessible from multiple machines.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (WHIRR-288) Add blob store persistence for
cluster state
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029198#comment-13029198 ]
Andrei Savu commented on WHIRR-288:
-----------------------------------
> Have you thought about using Avro/JSON as the serialization format? It would then be easier to evolve.
I believe the current format is good enough for now. I suggest we should change it later when the state data model evolves.
> Have a parser object that has public methods that are easier to test than the ones on ClusterStateStore. They don't really belong there as the ClusterStateStore could use any representation it chooses.
> Rather than use NoopClusterStateStore, the default might be in-memory (which you implemented elsewhere)?
I will address these two issues in one step later while improving code testability.
> Add blob store persistence for cluster state
> --------------------------------------------
>
> Key: WHIRR-288
> URL: https://issues.apache.org/jira/browse/WHIRR-288
> Project: Whirr
> Issue Type: Improvement
> Reporter: Andrei Savu
> Assignee: Andrei Savu
> Fix For: 0.5.0
>
> Attachments: WHIRR-288.patch, WHIRR-288.patch
>
>
> Right now Whirr persists the current cluster state (instance IDs, public and private IPs and roles) on the local filesystem (~/.whirr/<cluster-name>/instances). It should be possible to save the state as a blob so that it's easily accessible from multiple machines.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-288) Add blob store persistence for cluster
state
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-288:
------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I've just committed this. Guys thanks for helping and reviewing.
> Add blob store persistence for cluster state
> --------------------------------------------
>
> Key: WHIRR-288
> URL: https://issues.apache.org/jira/browse/WHIRR-288
> Project: Whirr
> Issue Type: Improvement
> Reporter: Andrei Savu
> Assignee: Andrei Savu
> Fix For: 0.5.0
>
> Attachments: WHIRR-288.patch, WHIRR-288.patch, WHIRR-288.patch, WHIRR-288.patch
>
>
> Right now Whirr persists the current cluster state (instance IDs, public and private IPs and roles) on the local filesystem (~/.whirr/<cluster-name>/instances). It should be possible to save the state as a blob so that it's easily accessible from multiple machines.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (WHIRR-288) Add blob store persistence for
cluster state
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029221#comment-13029221 ]
Andrei Savu commented on WHIRR-288:
-----------------------------------
I want to commit this but unfortunately I'm still seeing some errors (even if the integration tests are passing):
{noformat}
endpoint.getHost() is null for https://whirr_test.s3.amazonaws.com/whirr-zookeeper
{noformat}
Any ideas? I will investigate more to get it fixed.
> Add blob store persistence for cluster state
> --------------------------------------------
>
> Key: WHIRR-288
> URL: https://issues.apache.org/jira/browse/WHIRR-288
> Project: Whirr
> Issue Type: Improvement
> Reporter: Andrei Savu
> Assignee: Andrei Savu
> Fix For: 0.5.0
>
> Attachments: WHIRR-288.patch, WHIRR-288.patch
>
>
> Right now Whirr persists the current cluster state (instance IDs, public and private IPs and roles) on the local filesystem (~/.whirr/<cluster-name>/instances). It should be possible to save the state as a blob so that it's easily accessible from multiple machines.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-288) Add blob store persistence for cluster
state
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-288:
------------------------------
Attachment: WHIRR-288.patch
I was using an invalid container name. I've added code for strict format checking (thanks Adrian for help) and I've been able to start a cluster using blob store state persistence. I'm going to commit this.
> Add blob store persistence for cluster state
> --------------------------------------------
>
> Key: WHIRR-288
> URL: https://issues.apache.org/jira/browse/WHIRR-288
> Project: Whirr
> Issue Type: Improvement
> Reporter: Andrei Savu
> Assignee: Andrei Savu
> Fix For: 0.5.0
>
> Attachments: WHIRR-288.patch, WHIRR-288.patch, WHIRR-288.patch, WHIRR-288.patch
>
>
> Right now Whirr persists the current cluster state (instance IDs, public and private IPs and roles) on the local filesystem (~/.whirr/<cluster-name>/instances). It should be possible to save the state as a blob so that it's easily accessible from multiple machines.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (WHIRR-288) Add blob store persistence for
cluster state
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029119#comment-13029119 ]
Tom White commented on WHIRR-288:
---------------------------------
+1 Looks great.
A few comments, which we might address in follow-on issues.
* Have you thought about using Avro/JSON as the serialization format? It would then be easier to evolve.
* Have a parser object that has public methods that are easier to test than the ones on ClusterStateStore. They don't really belong there as the ClusterStateStore could use any representation it chooses.
* Rather than use NoopClusterStateStore, the default might be in-memory (which you implemented elsewhere)?
> Add blob store persistence for cluster state
> --------------------------------------------
>
> Key: WHIRR-288
> URL: https://issues.apache.org/jira/browse/WHIRR-288
> Project: Whirr
> Issue Type: Improvement
> Reporter: Andrei Savu
> Assignee: Andrei Savu
> Fix For: 0.5.0
>
> Attachments: WHIRR-288.patch, WHIRR-288.patch
>
>
> Right now Whirr persists the current cluster state (instance IDs, public and private IPs and roles) on the local filesystem (~/.whirr/<cluster-name>/instances). It should be possible to save the state as a blob so that it's easily accessible from multiple machines.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-288) Add blob store persistence for cluster
state
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-288:
------------------------------
Attachment: WHIRR-288.patch
Not tested (yet) patch (it still needs integration tests). Let me know what you think.
> Add blob store persistence for cluster state
> --------------------------------------------
>
> Key: WHIRR-288
> URL: https://issues.apache.org/jira/browse/WHIRR-288
> Project: Whirr
> Issue Type: Improvement
> Reporter: Andrei Savu
> Assignee: Andrei Savu
> Fix For: 0.5.0
>
> Attachments: WHIRR-288.patch
>
>
> Right now Whirr persists the current cluster state (instance IDs, public and private IPs and roles) on the local filesystem (~/.whirr/<cluster-name>/instances). It should be possible to save the state as a blob so that it's easily accessible from multiple machines.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-288) Add blob store persistence for cluster
state
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-288:
------------------------------
Status: Patch Available (was: Open)
> Add blob store persistence for cluster state
> --------------------------------------------
>
> Key: WHIRR-288
> URL: https://issues.apache.org/jira/browse/WHIRR-288
> Project: Whirr
> Issue Type: Improvement
> Reporter: Andrei Savu
> Assignee: Andrei Savu
> Fix For: 0.5.0
>
> Attachments: WHIRR-288.patch, WHIRR-288.patch
>
>
> Right now Whirr persists the current cluster state (instance IDs, public and private IPs and roles) on the local filesystem (~/.whirr/<cluster-name>/instances). It should be possible to save the state as a blob so that it's easily accessible from multiple machines.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-288) Add blob store persistence for cluster
state
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-288:
------------------------------
Attachment: WHIRR-288.patch
Added integration test and it works on EC2.
> Add blob store persistence for cluster state
> --------------------------------------------
>
> Key: WHIRR-288
> URL: https://issues.apache.org/jira/browse/WHIRR-288
> Project: Whirr
> Issue Type: Improvement
> Reporter: Andrei Savu
> Assignee: Andrei Savu
> Fix For: 0.5.0
>
> Attachments: WHIRR-288.patch, WHIRR-288.patch
>
>
> Right now Whirr persists the current cluster state (instance IDs, public and private IPs and roles) on the local filesystem (~/.whirr/<cluster-name>/instances). It should be possible to save the state as a blob so that it's easily accessible from multiple machines.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (WHIRR-288) Add blob store persistence for
cluster state
Posted by "Adrian Cole (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029333#comment-13029333 ]
Adrian Cole commented on WHIRR-288:
-----------------------------------
+1
> Add blob store persistence for cluster state
> --------------------------------------------
>
> Key: WHIRR-288
> URL: https://issues.apache.org/jira/browse/WHIRR-288
> Project: Whirr
> Issue Type: Improvement
> Reporter: Andrei Savu
> Assignee: Andrei Savu
> Fix For: 0.5.0
>
> Attachments: WHIRR-288.patch, WHIRR-288.patch, WHIRR-288.patch, WHIRR-288.patch
>
>
> Right now Whirr persists the current cluster state (instance IDs, public and private IPs and roles) on the local filesystem (~/.whirr/<cluster-name>/instances). It should be possible to save the state as a blob so that it's easily accessible from multiple machines.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-288) Add blob store persistence for cluster
state
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-288:
------------------------------
Attachment: WHIRR-288.patch
Updated patch with some fixes. Unfortunately it still doesn't work. Adrian, any ideas?
> Add blob store persistence for cluster state
> --------------------------------------------
>
> Key: WHIRR-288
> URL: https://issues.apache.org/jira/browse/WHIRR-288
> Project: Whirr
> Issue Type: Improvement
> Reporter: Andrei Savu
> Assignee: Andrei Savu
> Fix For: 0.5.0
>
> Attachments: WHIRR-288.patch, WHIRR-288.patch, WHIRR-288.patch
>
>
> Right now Whirr persists the current cluster state (instance IDs, public and private IPs and roles) on the local filesystem (~/.whirr/<cluster-name>/instances). It should be possible to save the state as a blob so that it's easily accessible from multiple machines.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira