You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by Alejandro Bonilla <ab...@linuxwireless.org> on 2014/05/02 03:42:25 UTC
Re: MongoDB cartridge with auto-scaling on Apache Stratos
Hi Akila,
On Mon, Apr 21, 2014 at 8:38 AM, Akila Ravihansa Perera
<ra...@wso2.com> wrote:
> Hi,
>
> I'm working on creating a new cartridge for mongodb with auto-scaling
> support. So far 3 use-cases have been identified so far.
>
> 1. mongodb scales horizontally (mongodb with sharding cluster)
> 2. mongodb scales vertically (mongodb with replica cluster)
> 3. mongodb replicated sharding cluster
Cool, you can make a cartridge using susestudio.com if you want, it
will take you 5 minutes. If you want one, I can provide it built
already.
> I have successfully deployed mongodb single node instance as a Stratos
> cartridge by using this Puppet script [1]. However, implementing
> auto-scaling for the mongodb cartridge requires some work.
>
> For a start, I'll be working on the first use-case scenario (mongodb with
> sharding cluster). Following is a brief work plan.
>
> 1. mongos (query routers) and config servers will be bundled in a single
> "mongolb" instance (mongo load balancer) which will be subscribed
> automatically by Stratos as the load balancer.
> 2. mongod servers (shard servers) will be bundled as "mongodb" instances
> which will get scaled up or down.
> 3. When an instance is spawned, it will get added to the mongos cluster by a
> Puppet script.
> Note - When implementing use-case scenario 2 and 3, Stratos Agent will have
> to check the member list and depending on the member count, initialize the
> replication set (master node) OR add the instance to the appropriate
> replication set.
This is good stuff and how it should work, yet, I've been in touch
with Mongo architects and they claim few or no real customers use
auto-scaling :( - they do run into scalability issues and query
balancing race conditions since even if their transactions are atomic,
it does not guarantee the other replica sets will already have the
data. Also, scaling down servers may lead to weirdness depending on
the complexity of the application.
Nevertheless, eventually this will take some traction and would be
interesting...
>
> Pl let me know your thoughts regarding this.
>
> [1] - https://github.com/echocat/puppet-mongodb
>
> Thanks.
>
> --
> Akila Ravihansa Perera
> Software Engineer
> WSO2 Inc.
> http://wso2.com
>
> Phone: +94 77 64 154 38
> Blog: http://ravihansa3000.blogspot.com
Re: MongoDB cartridge with auto-scaling on Apache Stratos
Posted by Alejandro Bonilla <ab...@linuxwireless.org>.
Akila,
That is all setup in the /etc/mongodb.conf or by starting mongodb with
the parameters. There can be 2 methods. 1. The image is all configured
and it just needs to spin up. or; 2. A PaaS is used that "injects"
the configuration either in the config file or loads mongo via the
command line with the needed parameters.
On Fri, May 2, 2014 at 12:51 AM, Akila Ravihansa Perera
<ra...@wso2.com> wrote:
> Hi Alejandro,
>
> Thanks a lot for the comments. I noticed that susestudio already has a
> MongoDB cartridge in their store. But I was not able to download it, got
> some server error message. This is without auto-scaling support I believe?
>
> My main aim is to provide replication with sharding support for MongoDB
> cartridge in Stratos. I agree that it will run into lot of weird behavior of
> applications when scaling down. What features do you think should a PaaS
> framework provide for a data cartridge like MongoDB? Really appreciate your
> thoughts regarding this.
>
> Thanks.
>
> On 2 May 2014 07:12, "Alejandro Bonilla" <ab...@linuxwireless.org> wrote:
>>
>> Hi Akila,
>>
>> On Mon, Apr 21, 2014 at 8:38 AM, Akila Ravihansa Perera
>> <ra...@wso2.com> wrote:
>> > Hi,
>> >
>> > I'm working on creating a new cartridge for mongodb with auto-scaling
>> > support. So far 3 use-cases have been identified so far.
>> >
>> > 1. mongodb scales horizontally (mongodb with sharding cluster)
>> > 2. mongodb scales vertically (mongodb with replica cluster)
>> > 3. mongodb replicated sharding cluster
>>
>> Cool, you can make a cartridge using susestudio.com if you want, it
>> will take you 5 minutes. If you want one, I can provide it built
>> already.
>>
>> > I have successfully deployed mongodb single node instance as a Stratos
>> > cartridge by using this Puppet script [1]. However, implementing
>> > auto-scaling for the mongodb cartridge requires some work.
>> >
>> > For a start, I'll be working on the first use-case scenario (mongodb
>> > with
>> > sharding cluster). Following is a brief work plan.
>> >
>> > 1. mongos (query routers) and config servers will be bundled in a single
>> > "mongolb" instance (mongo load balancer) which will be subscribed
>> > automatically by Stratos as the load balancer.
>> > 2. mongod servers (shard servers) will be bundled as "mongodb" instances
>> > which will get scaled up or down.
>> > 3. When an instance is spawned, it will get added to the mongos cluster
>> > by a
>> > Puppet script.
>> > Note - When implementing use-case scenario 2 and 3, Stratos Agent will
>> > have
>> > to check the member list and depending on the member count, initialize
>> > the
>> > replication set (master node) OR add the instance to the appropriate
>> > replication set.
>>
>> This is good stuff and how it should work, yet, I've been in touch
>> with Mongo architects and they claim few or no real customers use
>> auto-scaling :( - they do run into scalability issues and query
>> balancing race conditions since even if their transactions are atomic,
>> it does not guarantee the other replica sets will already have the
>> data. Also, scaling down servers may lead to weirdness depending on
>> the complexity of the application.
>>
>> Nevertheless, eventually this will take some traction and would be
>> interesting...
>>
>> >
>> > Pl let me know your thoughts regarding this.
>> >
>> > [1] - https://github.com/echocat/puppet-mongodb
>> >
>> > Thanks.
>> >
>> > --
>> > Akila Ravihansa Perera
>> > Software Engineer
>> > WSO2 Inc.
>> > http://wso2.com
>> >
>> > Phone: +94 77 64 154 38
>> > Blog: http://ravihansa3000.blogspot.com
Re: MongoDB cartridge with auto-scaling on Apache Stratos
Posted by Akila Ravihansa Perera <ra...@wso2.com>.
Hi Alejandro,
Thanks a lot for the comments. I noticed that susestudio already has a
MongoDB cartridge in their store. But I was not able to download it, got
some server error message. This is without auto-scaling support I believe?
My main aim is to provide replication with sharding support for MongoDB
cartridge in Stratos. I agree that it will run into lot of weird behavior
of applications when scaling down. What features do you think should a PaaS
framework provide for a data cartridge like MongoDB? Really appreciate your
thoughts regarding this.
Thanks.
On 2 May 2014 07:12, "Alejandro Bonilla" <ab...@linuxwireless.org> wrote:
> Hi Akila,
>
> On Mon, Apr 21, 2014 at 8:38 AM, Akila Ravihansa Perera
> <ra...@wso2.com> wrote:
> > Hi,
> >
> > I'm working on creating a new cartridge for mongodb with auto-scaling
> > support. So far 3 use-cases have been identified so far.
> >
> > 1. mongodb scales horizontally (mongodb with sharding cluster)
> > 2. mongodb scales vertically (mongodb with replica cluster)
> > 3. mongodb replicated sharding cluster
>
> Cool, you can make a cartridge using susestudio.com if you want, it
> will take you 5 minutes. If you want one, I can provide it built
> already.
>
> > I have successfully deployed mongodb single node instance as a Stratos
> > cartridge by using this Puppet script [1]. However, implementing
> > auto-scaling for the mongodb cartridge requires some work.
> >
> > For a start, I'll be working on the first use-case scenario (mongodb with
> > sharding cluster). Following is a brief work plan.
> >
> > 1. mongos (query routers) and config servers will be bundled in a single
> > "mongolb" instance (mongo load balancer) which will be subscribed
> > automatically by Stratos as the load balancer.
> > 2. mongod servers (shard servers) will be bundled as "mongodb" instances
> > which will get scaled up or down.
> > 3. When an instance is spawned, it will get added to the mongos cluster
> by a
> > Puppet script.
> > Note - When implementing use-case scenario 2 and 3, Stratos Agent will
> have
> > to check the member list and depending on the member count, initialize
> the
> > replication set (master node) OR add the instance to the appropriate
> > replication set.
>
> This is good stuff and how it should work, yet, I've been in touch
> with Mongo architects and they claim few or no real customers use
> auto-scaling :( - they do run into scalability issues and query
> balancing race conditions since even if their transactions are atomic,
> it does not guarantee the other replica sets will already have the
> data. Also, scaling down servers may lead to weirdness depending on
> the complexity of the application.
>
> Nevertheless, eventually this will take some traction and would be
> interesting...
>
> >
> > Pl let me know your thoughts regarding this.
> >
> > [1] - https://github.com/echocat/puppet-mongodb
> >
> > Thanks.
> >
> > --
> > Akila Ravihansa Perera
> > Software Engineer
> > WSO2 Inc.
> > http://wso2.com
> >
> > Phone: +94 77 64 154 38
> > Blog: http://ravihansa3000.blogspot.com
>
Re: MongoDB cartridge with auto-scaling on Apache Stratos
Posted by Lakmal Warusawithana <la...@wso2.com>.
On Fri, May 2, 2014 at 7:12 AM, Alejandro Bonilla <
abonilla@linuxwireless.org> wrote:
> Hi Akila,
>
> On Mon, Apr 21, 2014 at 8:38 AM, Akila Ravihansa Perera
> <ra...@wso2.com> wrote:
> > Hi,
> >
> > I'm working on creating a new cartridge for mongodb with auto-scaling
> > support. So far 3 use-cases have been identified so far.
> >
> > 1. mongodb scales horizontally (mongodb with sharding cluster)
> > 2. mongodb scales vertically (mongodb with replica cluster)
> > 3. mongodb replicated sharding cluster
>
> Cool, you can make a cartridge using susestudio.com if you want, it
> will take you 5 minutes. If you want one, I can provide it built
> already.
>
> > I have successfully deployed mongodb single node instance as a Stratos
> > cartridge by using this Puppet script [1]. However, implementing
> > auto-scaling for the mongodb cartridge requires some work.
> >
> > For a start, I'll be working on the first use-case scenario (mongodb with
> > sharding cluster). Following is a brief work plan.
> >
> > 1. mongos (query routers) and config servers will be bundled in a single
> > "mongolb" instance (mongo load balancer) which will be subscribed
> > automatically by Stratos as the load balancer.
> > 2. mongod servers (shard servers) will be bundled as "mongodb" instances
> > which will get scaled up or down.
> > 3. When an instance is spawned, it will get added to the mongos cluster
> by a
> > Puppet script.
> > Note - When implementing use-case scenario 2 and 3, Stratos Agent will
> have
> > to check the member list and depending on the member count, initialize
> the
> > replication set (master node) OR add the instance to the appropriate
> > replication set.
>
> This is good stuff and how it should work, yet, I've been in touch
> with Mongo architects and they claim few or no real customers use
> auto-scaling :( - they do run into scalability issues and query
> balancing race conditions since even if their transactions are atomic,
> it does not guarantee the other replica sets will already have the
> data. Also, scaling down servers may lead to weirdness depending on
> the complexity of the application.
>
> Nevertheless, eventually this will take some traction and would be
> interesting...
>
>
Interesting.., will make our hand dirty on this and see what are the
issues. Thanks Alejandro for finding, we may need to get some help form
Mongo architects when we get there.
> >
> > Pl let me know your thoughts regarding this.
> >
> > [1] - https://github.com/echocat/puppet-mongodb
> >
> > Thanks.
> >
> > --
> > Akila Ravihansa Perera
> > Software Engineer
> > WSO2 Inc.
> > http://wso2.com
> >
> > Phone: +94 77 64 154 38
> > Blog: http://ravihansa3000.blogspot.com
>
--
Lakmal Warusawithana
Director - Cloud Architecture; WSO2 Inc.
Mobile : +94714289692
Blog : http://lakmalsview.blogspot.com/