You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Eric LEMOINE <el...@mirantis.com> on 2015/12/21 10:11:44 UTC
mesos-elasticsearch vs Elasticsearch with Marathon
Hi
I am new to Mesos and I have a naive question related to Elasticsearch,
Mesos and Marathon.
So there's the mesos-elasticsearch [*] project which provides a Mesos
framework/scheduler for Elasticsearch. I guess it's also possible to run
Elasticsearch with Marathon. What are the fundamental differences between
the two approaches? When should one favor one approach over the other one?
What are the reasons for using mesos-elasticsearch instead of just running
Elasticsearch on top of Marathon?
Thanks for any insight.
[*] <http://mesos-elasticsearch.readthedocs.org>
Re: mesos-elasticsearch vs Elasticsearch with Marathon
Posted by tommy xiao <xi...@gmail.com>.
have a try on two ways. write some experience and share to us. it 's welcome
2015-12-21 17:11 GMT+08:00 Eric LEMOINE <el...@mirantis.com>:
> Hi
>
> I am new to Mesos and I have a naive question related to Elasticsearch,
> Mesos and Marathon.
>
> So there's the mesos-elasticsearch [*] project which provides a Mesos
> framework/scheduler for Elasticsearch. I guess it's also possible to run
> Elasticsearch with Marathon. What are the fundamental differences between
> the two approaches? When should one favor one approach over the other one?
> What are the reasons for using mesos-elasticsearch instead of just running
> Elasticsearch on top of Marathon?
>
> Thanks for any insight.
>
> [*] <http://mesos-elasticsearch.readthedocs.org>
>
--
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com
Re: mesos-elasticsearch vs Elasticsearch with Marathon
Posted by John Omernik <jo...@omernik.com>.
I'll just toss another way.. there is an elastic search on yarn framework
that actually works really nice with Apache Myriad (for running Yarn on
Mesos) I know it sounds a bit convoluted but I have set it up so I can
create ES clusters on demand, just give me a cluster name, and node size,
and it will spin up a cluster. Want to add more nodes? Great, each "scale"
operation adds another yarn application... so you could start with a 3 node
cluster, then add 2 nodes, add 2 nodes add 3 nodes. You'd have a 10 node
cluster, but you could scale down by 3 or 2 nodes because each time you've
added nodes it's a separate yarn application that can be killed.
I was using MapR FS as my base filesystem, so that also helps with data
storage etc.
On Mon, Dec 21, 2015 at 3:11 AM, Eric LEMOINE <el...@mirantis.com> wrote:
> Hi
>
> I am new to Mesos and I have a naive question related to Elasticsearch,
> Mesos and Marathon.
>
> So there's the mesos-elasticsearch [*] project which provides a Mesos
> framework/scheduler for Elasticsearch. I guess it's also possible to run
> Elasticsearch with Marathon. What are the fundamental differences between
> the two approaches? When should one favor one approach over the other one?
> What are the reasons for using mesos-elasticsearch instead of just running
> Elasticsearch on top of Marathon?
>
> Thanks for any insight.
>
> [*] <http://mesos-elasticsearch.readthedocs.org>
>
Re: mesos-elasticsearch vs Elasticsearch with Marathon
Posted by craig w <co...@gmail.com>.
We have elasticsearch in a docker container. We configure elasticsearch to
have ports 9200 and other port 9300 (this is for "transport.tcp.port"). We
modify the elasticsearch configuration file on startup to add the IP
addresses of other elasticsearch instances (we query mesos-dns for these ip
addresses).
On Mon, Dec 28, 2015 at 5:51 PM, Alex Rukletsov <al...@mesosphere.com> wrote:
> Craig,
>
> mind elaborating, how exactly do you run elasticsearch in Marathon?
>
> On Mon, Dec 28, 2015 at 8:36 PM, craig w <co...@gmail.com> wrote:
> > In terms of discovery, elasticsearch provides that out of the box
> >
> https://www.elastic.co/guide/en/elasticsearch/reference/1.4/modules-discovery.html
> .
> > We deploy elasticsearch via Marathon and it works great.
> >
> > On Mon, Dec 28, 2015 at 2:17 PM, Eric LEMOINE <el...@mirantis.com>
> wrote:
> >>
> >> On Mon, Dec 28, 2015 at 7:55 PM, Alex Rukletsov <al...@mesosphere.com>
> >> wrote:
> >> > Eric—
> >> >
> >> > give me a chance to answer that before you fall into frustration : ).
> >> > Also, you can directly write to framework developers
> >> > (mesos-es@container-solutions.com) and they either confirm or bust my
> >> > guess. Or maybe one of the authors — Frank — will chime in in this
> >> > thread.
> >> >
> >> > Marathon has no idea about application logic, hence a "scale"
> >> > operation just starts more application instances. But sometimes you
> >> > may want to do extra job (track instances, report ip:port of a new
> >> > instance to existing instances, and so on). That's when a dedicated
> >> > framework makes sense. Each framework has a scheduler that is able to
> >> > track each instance and do all aforementioned actions.
> >> >
> >> > How this maps to your question? AFAIK, all Elasticsearch nodes should
> >> > see each other, hence once a new node is started, it should be somehow
> >> > advertised to other nodes. You can do it by wrapping Elasticsearch
> >> > command in a shell script and maintain some sort of an out-of-band
> >> > registry, take a look at one of the first efforts [1] to run
> >> > Elasticsearch on Mesos to get an impression how it may look like. But
> >> > you can use a dedicated framework instead : ).
> >> >
> >> > [1] https://github.com/mesosphere/elasticsearch-mesos
> >>
> >>
> >> That makes great sense Alex. Thanks for chiming in.
> >
> >
> >
> >
> > --
> >
> > https://github.com/mindscratch
> > https://www.google.com/+CraigWickesser
> > https://twitter.com/mind_scratch
> > https://twitter.com/craig_links
>
--
https://github.com/mindscratch
https://www.google.com/+CraigWickesser
https://twitter.com/mind_scratch
https://twitter.com/craig_links
Re: mesos-elasticsearch vs Elasticsearch with Marathon
Posted by Alex Rukletsov <al...@mesosphere.com>.
Craig,
mind elaborating, how exactly do you run elasticsearch in Marathon?
On Mon, Dec 28, 2015 at 8:36 PM, craig w <co...@gmail.com> wrote:
> In terms of discovery, elasticsearch provides that out of the box
> https://www.elastic.co/guide/en/elasticsearch/reference/1.4/modules-discovery.html.
> We deploy elasticsearch via Marathon and it works great.
>
> On Mon, Dec 28, 2015 at 2:17 PM, Eric LEMOINE <el...@mirantis.com> wrote:
>>
>> On Mon, Dec 28, 2015 at 7:55 PM, Alex Rukletsov <al...@mesosphere.com>
>> wrote:
>> > Eric—
>> >
>> > give me a chance to answer that before you fall into frustration : ).
>> > Also, you can directly write to framework developers
>> > (mesos-es@container-solutions.com) and they either confirm or bust my
>> > guess. Or maybe one of the authors — Frank — will chime in in this
>> > thread.
>> >
>> > Marathon has no idea about application logic, hence a "scale"
>> > operation just starts more application instances. But sometimes you
>> > may want to do extra job (track instances, report ip:port of a new
>> > instance to existing instances, and so on). That's when a dedicated
>> > framework makes sense. Each framework has a scheduler that is able to
>> > track each instance and do all aforementioned actions.
>> >
>> > How this maps to your question? AFAIK, all Elasticsearch nodes should
>> > see each other, hence once a new node is started, it should be somehow
>> > advertised to other nodes. You can do it by wrapping Elasticsearch
>> > command in a shell script and maintain some sort of an out-of-band
>> > registry, take a look at one of the first efforts [1] to run
>> > Elasticsearch on Mesos to get an impression how it may look like. But
>> > you can use a dedicated framework instead : ).
>> >
>> > [1] https://github.com/mesosphere/elasticsearch-mesos
>>
>>
>> That makes great sense Alex. Thanks for chiming in.
>
>
>
>
> --
>
> https://github.com/mindscratch
> https://www.google.com/+CraigWickesser
> https://twitter.com/mind_scratch
> https://twitter.com/craig_links
Re: mesos-elasticsearch vs Elasticsearch with Marathon
Posted by craig w <co...@gmail.com>.
In terms of discovery, elasticsearch provides that out of the box
https://www.elastic.co/guide/en/elasticsearch/reference/1.4/modules-discovery.html.
We deploy elasticsearch via Marathon and it works great.
On Mon, Dec 28, 2015 at 2:17 PM, Eric LEMOINE <el...@mirantis.com> wrote:
> On Mon, Dec 28, 2015 at 7:55 PM, Alex Rukletsov <al...@mesosphere.com>
> wrote:
> > Eric—
> >
> > give me a chance to answer that before you fall into frustration : ).
> > Also, you can directly write to framework developers
> > (mesos-es@container-solutions.com) and they either confirm or bust my
> > guess. Or maybe one of the authors — Frank — will chime in in this
> > thread.
> >
> > Marathon has no idea about application logic, hence a "scale"
> > operation just starts more application instances. But sometimes you
> > may want to do extra job (track instances, report ip:port of a new
> > instance to existing instances, and so on). That's when a dedicated
> > framework makes sense. Each framework has a scheduler that is able to
> > track each instance and do all aforementioned actions.
> >
> > How this maps to your question? AFAIK, all Elasticsearch nodes should
> > see each other, hence once a new node is started, it should be somehow
> > advertised to other nodes. You can do it by wrapping Elasticsearch
> > command in a shell script and maintain some sort of an out-of-band
> > registry, take a look at one of the first efforts [1] to run
> > Elasticsearch on Mesos to get an impression how it may look like. But
> > you can use a dedicated framework instead : ).
> >
> > [1] https://github.com/mesosphere/elasticsearch-mesos
>
>
> That makes great sense Alex. Thanks for chiming in.
>
--
https://github.com/mindscratch
https://www.google.com/+CraigWickesser
https://twitter.com/mind_scratch
https://twitter.com/craig_links
Re: mesos-elasticsearch vs Elasticsearch with Marathon
Posted by Eric LEMOINE <el...@mirantis.com>.
On Mon, Dec 28, 2015 at 7:55 PM, Alex Rukletsov <al...@mesosphere.com> wrote:
> Eric—
>
> give me a chance to answer that before you fall into frustration : ).
> Also, you can directly write to framework developers
> (mesos-es@container-solutions.com) and they either confirm or bust my
> guess. Or maybe one of the authors — Frank — will chime in in this
> thread.
>
> Marathon has no idea about application logic, hence a "scale"
> operation just starts more application instances. But sometimes you
> may want to do extra job (track instances, report ip:port of a new
> instance to existing instances, and so on). That's when a dedicated
> framework makes sense. Each framework has a scheduler that is able to
> track each instance and do all aforementioned actions.
>
> How this maps to your question? AFAIK, all Elasticsearch nodes should
> see each other, hence once a new node is started, it should be somehow
> advertised to other nodes. You can do it by wrapping Elasticsearch
> command in a shell script and maintain some sort of an out-of-band
> registry, take a look at one of the first efforts [1] to run
> Elasticsearch on Mesos to get an impression how it may look like. But
> you can use a dedicated framework instead : ).
>
> [1] https://github.com/mesosphere/elasticsearch-mesos
That makes great sense Alex. Thanks for chiming in.
Re: mesos-elasticsearch vs Elasticsearch with Marathon
Posted by Alex Rukletsov <al...@mesosphere.com>.
Eric—
give me a chance to answer that before you fall into frustration : ).
Also, you can directly write to framework developers
(mesos-es@container-solutions.com) and they either confirm or bust my
guess. Or maybe one of the authors — Frank — will chime in in this
thread.
Marathon has no idea about application logic, hence a "scale"
operation just starts more application instances. But sometimes you
may want to do extra job (track instances, report ip:port of a new
instance to existing instances, and so on). That's when a dedicated
framework makes sense. Each framework has a scheduler that is able to
track each instance and do all aforementioned actions.
How this maps to your question? AFAIK, all Elasticsearch nodes should
see each other, hence once a new node is started, it should be somehow
advertised to other nodes. You can do it by wrapping Elasticsearch
command in a shell script and maintain some sort of an out-of-band
registry, take a look at one of the first efforts [1] to run
Elasticsearch on Mesos to get an impression how it may look like. But
you can use a dedicated framework instead : ).
[1] https://github.com/mesosphere/elasticsearch-mesos
On Wed, Dec 23, 2015 at 10:30 AM, Eric LEMOINE <el...@mirantis.com> wrote:
> On Tue, Dec 22, 2015 at 10:05 AM, craig w <co...@gmail.com> wrote:
>> We'd like to use the framework once some more features are available (see
>> the road map).
>>
>> Currently we deploy ES in docker using marathon.
>
>
>
> Thank you all for your responses. I get that the situation is not as
> clear as I expected :)
Re: mesos-elasticsearch vs Elasticsearch with Marathon
Posted by Eric LEMOINE <el...@mirantis.com>.
On Tue, Dec 22, 2015 at 10:05 AM, craig w <co...@gmail.com> wrote:
> We'd like to use the framework once some more features are available (see
> the road map).
>
> Currently we deploy ES in docker using marathon.
Thank you all for your responses. I get that the situation is not as
clear as I expected :)
Re: mesos-elasticsearch vs Elasticsearch with Marathon
Posted by craig w <co...@gmail.com>.
We'd like to use the framework once some more features are available (see
the road map).
Currently we deploy ES in docker using marathon.
On Dec 21, 2015 11:14 PM, "Weitao" <zh...@gmail.com> wrote:
> IMO, es as a framework running on Mesos is our future, atst, we can deploy
> the es framework over Marathon.
>
> 在 2015年12月21日，17:11，Eric LEMOINE <el...@mirantis.com> 写道：
>
> Hi
>
> I am new to Mesos and I have a naive question related to Elasticsearch,
> Mesos and Marathon.
>
> So there's the mesos-elasticsearch [*] project which provides a Mesos
> framework/scheduler for Elasticsearch. I guess it's also possible to run
> Elasticsearch with Marathon. What are the fundamental differences between
> the two approaches? When should one favor one approach over the other one?
> What are the reasons for using mesos-elasticsearch instead of just running
> Elasticsearch on top of Marathon?
>
> Thanks for any insight.
>
> [*] <http://mesos-elasticsearch.readthedocs.org>
>
>
Re: mesos-elasticsearch vs Elasticsearch with Marathon
Posted by Weitao <zh...@gmail.com>.
IMO, es as a framework running on Mesos is our future, atst, we can deploy the es framework over Marathon.
> �� 2015��12��21�գ�17:11��Eric LEMOINE <el...@mirantis.com> д����
>
> Hi
>
> I am new to Mesos and I have a naive question related to Elasticsearch, Mesos and Marathon.
>
> So there's the mesos-elasticsearch [*] project which provides a Mesos framework/scheduler for Elasticsearch. I guess it's also possible to run Elasticsearch with Marathon. What are the fundamental differences between the two approaches? When should one favor one approach over the other one? What are the reasons for using mesos-elasticsearch instead of just running Elasticsearch on top of Marathon?
>
> Thanks for any insight.
>
> [*] <http://mesos-elasticsearch.readthedocs.org>