You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@whirr.apache.org by "Andrei Savu (JIRA)" <ji...@apache.org> on 2011/03/13 20:20:59 UTC
[jira] Created: (WHIRR-261) Add ElasticSearch as a service
Add ElasticSearch as a service
------------------------------
Key: WHIRR-261
URL: https://issues.apache.org/jira/browse/WHIRR-261
Project: Whirr
Issue Type: New Feature
Reporter: Andrei Savu
ElasticSearch is a distributed RESTful search engine built for the cloud.
[1] http://www.elasticsearch.org/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-261) Add ElasticSearch as a service
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-261:
------------------------------
Attachment: WHIRR-261.patch
Minor updates to patch:
* if not on EC2 configure unicast discovery (most of the cloud providers deny multicast)
* retry tarball download on failure
Integration tests are passing. I'm planning to commit this tomorrow. We can fix any remaining undiscovered issues in new JIRAs.
> Add ElasticSearch as a service
> ------------------------------
>
> Key: WHIRR-261
> URL: https://issues.apache.org/jira/browse/WHIRR-261
> Project: Whirr
> Issue Type: New Feature
> Reporter: Andrei Savu
> Assignee: Andrei Savu
> Fix For: 0.5.0
>
> Attachments: WHIRR-261.patch, WHIRR-261.patch, WHIRR-261.patch, WHIRR-261.patch, WHIRR-261.patch
>
>
> ElasticSearch is a distributed RESTful search engine built for the cloud.
> [1] http://www.elasticsearch.org/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (WHIRR-261) Add ElasticSearch as a service
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu reassigned WHIRR-261:
---------------------------------
Assignee: Andrei Savu
> Add ElasticSearch as a service
> ------------------------------
>
> Key: WHIRR-261
> URL: https://issues.apache.org/jira/browse/WHIRR-261
> Project: Whirr
> Issue Type: New Feature
> Reporter: Andrei Savu
> Assignee: Andrei Savu
> Attachments: WHIRR-261.patch, WHIRR-261.patch
>
>
> ElasticSearch is a distributed RESTful search engine built for the cloud.
> [1] http://www.elasticsearch.org/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-261) Add ElasticSearch as a service
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-261:
------------------------------
Fix Version/s: 0.5.0
> Add ElasticSearch as a service
> ------------------------------
>
> Key: WHIRR-261
> URL: https://issues.apache.org/jira/browse/WHIRR-261
> Project: Whirr
> Issue Type: New Feature
> Reporter: Andrei Savu
> Assignee: Andrei Savu
> Fix For: 0.5.0
>
> Attachments: WHIRR-261.patch, WHIRR-261.patch, WHIRR-261.patch
>
>
> ElasticSearch is a distributed RESTful search engine built for the cloud.
> [1] http://www.elasticsearch.org/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-261) Add ElasticSearch as a service
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-261:
------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I've just committed this. Thanks Adrian for reviewing.
> Add ElasticSearch as a service
> ------------------------------
>
> Key: WHIRR-261
> URL: https://issues.apache.org/jira/browse/WHIRR-261
> Project: Whirr
> Issue Type: New Feature
> Reporter: Andrei Savu
> Assignee: Andrei Savu
> Fix For: 0.5.0
>
> Attachments: WHIRR-261.patch, WHIRR-261.patch, WHIRR-261.patch, WHIRR-261.patch, WHIRR-261.patch
>
>
> ElasticSearch is a distributed RESTful search engine built for the cloud.
> [1] http://www.elasticsearch.org/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-261) Add ElasticSearch as a service
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-261:
------------------------------
Attachment: WHIRR-261.patch
Preliminary version of the patch: no tests, no configuration parameters.
> Add ElasticSearch as a service
> ------------------------------
>
> Key: WHIRR-261
> URL: https://issues.apache.org/jira/browse/WHIRR-261
> Project: Whirr
> Issue Type: New Feature
> Reporter: Andrei Savu
> Attachments: WHIRR-261.patch
>
>
> ElasticSearch is a distributed RESTful search engine built for the cloud.
> [1] http://www.elasticsearch.org/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-261) Add ElasticSearch as a service
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-261:
------------------------------
Status: Patch Available (was: Open)
> Add ElasticSearch as a service
> ------------------------------
>
> Key: WHIRR-261
> URL: https://issues.apache.org/jira/browse/WHIRR-261
> Project: Whirr
> Issue Type: New Feature
> Reporter: Andrei Savu
> Assignee: Andrei Savu
> Fix For: 0.5.0
>
> Attachments: WHIRR-261.patch, WHIRR-261.patch, WHIRR-261.patch, WHIRR-261.patch
>
>
> ElasticSearch is a distributed RESTful search engine built for the cloud.
> [1] http://www.elasticsearch.org/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (WHIRR-261) Add ElasticSearch as a service
Posted by "Shay Banon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010121#comment-13010121 ]
Shay Banon commented on WHIRR-261:
----------------------------------
Heya, not too familiar with whirr, so I did not exactly saw where the discovery part is configured? Does it use elasticsearch built in ec2 based discovery module, or does it build a unicast list of hosts for it to use?
Also, the default gateway is local, meaning that data is stored on the local/ebs drive. Of course, when on ec2, local drives are wiped, but, if on EBS, then it can restore its state from there.
> Add ElasticSearch as a service
> ------------------------------
>
> Key: WHIRR-261
> URL: https://issues.apache.org/jira/browse/WHIRR-261
> Project: Whirr
> Issue Type: New Feature
> Reporter: Andrei Savu
> Assignee: Andrei Savu
> Fix For: 0.5.0
>
> Attachments: WHIRR-261.patch, WHIRR-261.patch, WHIRR-261.patch
>
>
> ElasticSearch is a distributed RESTful search engine built for the cloud.
> [1] http://www.elasticsearch.org/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (WHIRR-261) Add ElasticSearch as a service
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011656#comment-13011656 ]
Andrei Savu commented on WHIRR-261:
-----------------------------------
I will submit a new version of this patch Monday with more improvements. Thanks for reviewing.
> Add ElasticSearch as a service
> ------------------------------
>
> Key: WHIRR-261
> URL: https://issues.apache.org/jira/browse/WHIRR-261
> Project: Whirr
> Issue Type: New Feature
> Reporter: Andrei Savu
> Assignee: Andrei Savu
> Fix For: 0.5.0
>
> Attachments: WHIRR-261.patch, WHIRR-261.patch, WHIRR-261.patch, WHIRR-261.patch
>
>
> ElasticSearch is a distributed RESTful search engine built for the cloud.
> [1] http://www.elasticsearch.org/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-261) Add ElasticSearch as a service
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-261:
------------------------------
Attachment: WHIRR-261.patch
This patch generates the configuration file using options from the cluster spec. I've done testing on AWS. The only remaining issue: support for cloudservers. I appreciate any feedback on this patch.
> Add ElasticSearch as a service
> ------------------------------
>
> Key: WHIRR-261
> URL: https://issues.apache.org/jira/browse/WHIRR-261
> Project: Whirr
> Issue Type: New Feature
> Reporter: Andrei Savu
> Assignee: Andrei Savu
> Attachments: WHIRR-261.patch, WHIRR-261.patch, WHIRR-261.patch
>
>
> ElasticSearch is a distributed RESTful search engine built for the cloud.
> [1] http://www.elasticsearch.org/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (WHIRR-261) Add ElasticSearch as a service
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010245#comment-13010245 ]
Andrei Savu commented on WHIRR-261:
-----------------------------------
By default Whirr will configure elasticsearch to use the EC2 discovery module on AWS and store the index in memory with no disk data persistence. On cloudservers I'm thinking about using unicast and build a list of hosts. All configuration parameters can be set in the cluster spec .properties file. A typical recipe for AWS looks like this:
{code}
# Change the cluster name here
whirr.cluster-name=elasticsearch
# This is also the elasticsearch cluster name
# Change the number of machines in the cluster here
whirr.instance-templates=2 elasticsearch
whirr.hardware-min-ram=2048
# For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
whirr.provider=aws-ec2
whirr.identity=${env:AWS_ACCESS_KEY_ID}
whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
# S3 persistence
es.gateway.type=s3
es.gateway.s3.bucket=elasticsearch
{code}
> Add ElasticSearch as a service
> ------------------------------
>
> Key: WHIRR-261
> URL: https://issues.apache.org/jira/browse/WHIRR-261
> Project: Whirr
> Issue Type: New Feature
> Reporter: Andrei Savu
> Assignee: Andrei Savu
> Fix For: 0.5.0
>
> Attachments: WHIRR-261.patch, WHIRR-261.patch, WHIRR-261.patch
>
>
> ElasticSearch is a distributed RESTful search engine built for the cloud.
> [1] http://www.elasticsearch.org/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-261) Add ElasticSearch as a service
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-261:
------------------------------
Attachment: WHIRR-261.patch
Tested with AWS and Rackspace. I'm happy with this patch version. Please review.
> Add ElasticSearch as a service
> ------------------------------
>
> Key: WHIRR-261
> URL: https://issues.apache.org/jira/browse/WHIRR-261
> Project: Whirr
> Issue Type: New Feature
> Reporter: Andrei Savu
> Assignee: Andrei Savu
> Fix For: 0.5.0
>
> Attachments: WHIRR-261.patch, WHIRR-261.patch, WHIRR-261.patch, WHIRR-261.patch
>
>
> ElasticSearch is a distributed RESTful search engine built for the cloud.
> [1] http://www.elasticsearch.org/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (WHIRR-261) Add ElasticSearch as a service
Posted by "Adrian Cole (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011582#comment-13011582 ]
Adrian Cole commented on WHIRR-261:
-----------------------------------
+1
looks pretty straightforward. We can probably make this more robust over time, esp as we add features to whirr such as topology changing and resilience. For now, seems a good start.
tested on aws-ec2 and cloudservers-us
> Add ElasticSearch as a service
> ------------------------------
>
> Key: WHIRR-261
> URL: https://issues.apache.org/jira/browse/WHIRR-261
> Project: Whirr
> Issue Type: New Feature
> Reporter: Andrei Savu
> Assignee: Andrei Savu
> Fix For: 0.5.0
>
> Attachments: WHIRR-261.patch, WHIRR-261.patch, WHIRR-261.patch, WHIRR-261.patch
>
>
> ElasticSearch is a distributed RESTful search engine built for the cloud.
> [1] http://www.elasticsearch.org/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (WHIRR-261) Add ElasticSearch as a service
Posted by "Andrei Savu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/WHIRR-261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrei Savu updated WHIRR-261:
------------------------------
Attachment: WHIRR-261.patch
Added integration test: check that all nodes join the cluster.
> Add ElasticSearch as a service
> ------------------------------
>
> Key: WHIRR-261
> URL: https://issues.apache.org/jira/browse/WHIRR-261
> Project: Whirr
> Issue Type: New Feature
> Reporter: Andrei Savu
> Attachments: WHIRR-261.patch, WHIRR-261.patch
>
>
> ElasticSearch is a distributed RESTful search engine built for the cloud.
> [1] http://www.elasticsearch.org/
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira