You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by Zaid Mohsin <za...@cloudsoftcorp.com> on 2014/11/03 15:44:45 UTC
MongoDBReplicaSetImpl error (out of memory)
Hi,
I’m trying to launch the MongoDB blueprint (‘mongo-sharded.yaml’) but I’m facing the following error. The cluster keeps on resizing ('Resize MongoDBReplicaSetImpl{id=UGwzKMYx} from 0 to 3’) calling start() multiple times until it runs out of memory:
Service is not up when setting running on MongoDBReplicaSetImpl{id=ABrSIbr2}; delayed 1s 377ms but Sensor: service.isUp (java.lang.Boolean) did not recover from false; not-up-indicators={service-not-up-indicators-from-children-and-members=MongoDBReplicaSetImpl{id=vAqS7FvJ} is not up}
2014-11-03 14:41:18,656 WARN Attempt #1/1 to obtain machine threw error: com.google.inject.internal.util.$ComputationException: com.google.inject.internal.util.$ComputationException: java.lang.OutOfMemoryError: PermGen space
2014-11-03 14:41:18,657 WARN Failed to create entity using spec EntitySpec{type=interface brooklyn.entity.nosql.mongodb.MongoDBReplicaSet} (rethrowing)
java.lang.OutOfMemoryError: PermGen space
Anyone know what’s the solution here?
Regards
Re: MongoDBReplicaSetImpl error (out of memory)
Posted by Zaid Mohsin <za...@cloudsoftcorp.com>.
Hi Aled, I’m running brooklyn through the IDE, but copying the blueprint in YAML format to Brooklyn’s web console.
when running the following blueprint:
name: Mongo3
services:
- serviceType: brooklyn.entity.nosql.mongodb.MongoDBReplicaSet
name: MongoDB Replica Set
brooklyn.config:
cluster.initial.size: 3
I get no errors. But when I run the sharded deployment i.e.:
services:
- serviceType: brooklyn.entity.nosql.mongodb.sharding.MongoDBShardedDeployment
id: shardeddeployment
name: Mongo Deploy
brooklyn.config:
initialRouterClusterSize: 1
initialShardClusterSize: 1
configClusterSize: 1
shardReplicaSetSize: 1
- serviceType: brooklyn.entity.nosql.mongodb.MongoDBClient
name: MongoDB Javascript Client
id: client
brooklyn.config:
shardedDeployment: $brooklyn:component("shardeddeployment")
scripts:
insert: classpath://mongo-product-insert.js
startupJsScripts: # These will be run in order on startup
- insert
I get many iterations of this message:
2014-11-03 17:16:15,752 INFO Resize MongoDBReplicaSetImpl{id=WUsXtVle} from 0 to 1
2014-11-03 17:16:15,783 INFO Resize MongoDBReplicaSetImpl{id=bdT9Hc6a} from 0 to 1
2014-11-03 17:16:15,844 INFO Resize MongoDBReplicaSetImpl{id=LyMiSt8W} from 0 to 1
2014-11-03 17:16:15,892 INFO Resize MongoDBReplicaSetImpl{id=rwZG4Ydw} from 0 to 1
2014-11-03 17:16:15,928 INFO Resize MongoDBReplicaSetImpl{id=EGtlEFfE} from 0 to 1
2014-11-03 17:16:15,961 INFO Resize MongoDBReplicaSetImpl{id=G8BPduDd} from 0 to 1
2014-11-03 17:16:15,987 INFO Resize MongoDBReplicaSetImpl{id=hYKf0uQB} from 0 to 1
2014-11-03 17:16:16,015 INFO Resize MongoDBReplicaSetImpl{id=Lyu1Vlth} from 0 to 1
until java runs out of memory.
Did that make it clearer?
Regards
> On 3 Nov 2014, at 17:08, Aled Sage <al...@gmail.com> wrote:
>
> Hi Zaid,
>
> How are you running this? Is it with the `brooklyn` script or via an IDE?
>
> Depending on how much you are planning to run in the Brooklyn instance, you'll want to launch it with something like:
>
> -Xms512m -Xmx512m -XX:MaxPermSize=256m
>
> Running out of PermGen means that it has run out of memory for loading classes and for interned strings (i.e. constants). The last of the arguments above increases the space available.
>
> If you are running with the `brooklyn` script, it should already be doing this.
>
> ---
> Not sure I followed your description: it keeps resizing for 0 to 3 multiple times? How many times has it tried? Is that with the same MongoDBReplicaSetImpl (id=UGwzKMYx) each time?
>
> Aled
>
>
> On 03/11/2014 14:44, Zaid Mohsin wrote:
>> Hi,
>> I’m trying to launch the MongoDB blueprint (‘mongo-sharded.yaml’) but I’m facing the following error. The cluster keeps on resizing ('Resize MongoDBReplicaSetImpl{id=UGwzKMYx} from 0 to 3’) calling start() multiple times until it runs out of memory:
>>
>> Service is not up when setting running on MongoDBReplicaSetImpl{id=ABrSIbr2}; delayed 1s 377ms but Sensor: service.isUp (java.lang.Boolean) did not recover from false; not-up-indicators={service-not-up-indicators-from-children-and-members=MongoDBReplicaSetImpl{id=vAqS7FvJ} is not up}
>>
>> 2014-11-03 14:41:18,656 WARN Attempt #1/1 to obtain machine threw error: com.google.inject.internal.util.$ComputationException: com.google.inject.internal.util.$ComputationException: java.lang.OutOfMemoryError: PermGen space
>>
>> 2014-11-03 14:41:18,657 WARN Failed to create entity using spec EntitySpec{type=interface brooklyn.entity.nosql.mongodb.MongoDBReplicaSet} (rethrowing)
>> java.lang.OutOfMemoryError: PermGen space
>>
>> Anyone know what’s the solution here?
>>
>> Regards
>
Re: MongoDBReplicaSetImpl error (out of memory)
Posted by Aled Sage <al...@gmail.com>.
Hi Zaid,
See https://github.com/apache/incubator-brooklyn/pull/293 for a fix.
That also includes a fix for the race causing occasional integration
test failures: if "adding the shard" to the router fails, then schedule
a retry rather than just logging the problem.
Aled
On 03/11/2014 22:15, Aled Sage wrote:
> Hi Zaid,
>
> This is a bug (a regression I introduced 6 days ago while fixing an
> integration test - sorry!)
>
> If you deploy that via the Brooklyn web-console and look at the app,
> you'll see the "Cluster of MongoDBReplicaSet" has a child that is a
> "Cluster of MongoDBReplicaSet", ad infinitum.
>
> The problem was introduced in MongoDBShardedDeploymentImpl.init where
> it sets the shard cluster's memberSpec to a replica set, but
> unfortunately that config is inherited by the replica set so its
> members are also replica sets!
>
> I'll look at fixing that now.
>
> Aled
>
>
> On 03/11/2014 17:19, Martin Harris wrote:
>> I think the real problem is that .start() is being called repeatedly. I
>> know there have been some changes to doStart() etc recently, so it
>> may be
>> related to that
>>
>> Cheers
>>
>> M
>>
>> On 3 November 2014 17:08, Aled Sage <al...@gmail.com> wrote:
>>
>>> Hi Zaid,
>>>
>>> How are you running this? Is it with the `brooklyn` script or via an
>>> IDE?
>>>
>>> Depending on how much you are planning to run in the Brooklyn instance,
>>> you'll want to launch it with something like:
>>>
>>> -Xms512m -Xmx512m -XX:MaxPermSize=256m
>>>
>>> Running out of PermGen means that it has run out of memory for loading
>>> classes and for interned strings (i.e. constants). The last of the
>>> arguments above increases the space available.
>>>
>>> If you are running with the `brooklyn` script, it should already be
>>> doing
>>> this.
>>>
>>> ---
>>> Not sure I followed your description: it keeps resizing for 0 to 3
>>> multiple times? How many times has it tried? Is that with the same
>>> MongoDBReplicaSetImpl (id=UGwzKMYx) each time?
>>>
>>> Aled
>>>
>>>
>>>
>>> On 03/11/2014 14:44, Zaid Mohsin wrote:
>>>
>>>> Hi,
>>>> I’m trying to launch the MongoDB blueprint (‘mongo-sharded.yaml’)
>>>> but I’m
>>>> facing the following error. The cluster keeps on resizing ('Resize
>>>> MongoDBReplicaSetImpl{id=UGwzKMYx} from 0 to 3’) calling start()
>>>> multiple times until it runs out of memory:
>>>>
>>>> Service is not up when setting running on
>>>> MongoDBReplicaSetImpl{id=ABrSIbr2};
>>>> delayed 1s 377ms but Sensor: service.isUp (java.lang.Boolean) did not
>>>> recover from false; not-up-indicators={service-not-up-indicators-from-
>>>> children-and-members=MongoDBReplicaSetImpl{id=vAqS7FvJ} is not up}
>>>>
>>>> 2014-11-03 14:41:18,656 WARN Attempt #1/1 to obtain machine threw
>>>> error:
>>>> com.google.inject.internal.util.$ComputationException:
>>>> com.google.inject.internal.util.$ComputationException:
>>>> java.lang.OutOfMemoryError: PermGen space
>>>>
>>>> 2014-11-03 14:41:18,657 WARN Failed to create entity using spec
>>>> EntitySpec{type=interface
>>>> brooklyn.entity.nosql.mongodb.MongoDBReplicaSet}
>>>> (rethrowing)
>>>> java.lang.OutOfMemoryError: PermGen space
>>>>
>>>> Anyone know what’s the solution here?
>>>>
>>>> Regards
>>>>
>>>
>>
>
Re: MongoDBReplicaSetImpl error (out of memory)
Posted by Aled Sage <al...@gmail.com>.
Hi Zaid,
This is a bug (a regression I introduced 6 days ago while fixing an
integration test - sorry!)
If you deploy that via the Brooklyn web-console and look at the app,
you'll see the "Cluster of MongoDBReplicaSet" has a child that is a
"Cluster of MongoDBReplicaSet", ad infinitum.
The problem was introduced in MongoDBShardedDeploymentImpl.init where it
sets the shard cluster's memberSpec to a replica set, but unfortunately
that config is inherited by the replica set so its members are also
replica sets!
I'll look at fixing that now.
Aled
On 03/11/2014 17:19, Martin Harris wrote:
> I think the real problem is that .start() is being called repeatedly. I
> know there have been some changes to doStart() etc recently, so it may be
> related to that
>
> Cheers
>
> M
>
> On 3 November 2014 17:08, Aled Sage <al...@gmail.com> wrote:
>
>> Hi Zaid,
>>
>> How are you running this? Is it with the `brooklyn` script or via an IDE?
>>
>> Depending on how much you are planning to run in the Brooklyn instance,
>> you'll want to launch it with something like:
>>
>> -Xms512m -Xmx512m -XX:MaxPermSize=256m
>>
>> Running out of PermGen means that it has run out of memory for loading
>> classes and for interned strings (i.e. constants). The last of the
>> arguments above increases the space available.
>>
>> If you are running with the `brooklyn` script, it should already be doing
>> this.
>>
>> ---
>> Not sure I followed your description: it keeps resizing for 0 to 3
>> multiple times? How many times has it tried? Is that with the same
>> MongoDBReplicaSetImpl (id=UGwzKMYx) each time?
>>
>> Aled
>>
>>
>>
>> On 03/11/2014 14:44, Zaid Mohsin wrote:
>>
>>> Hi,
>>> I’m trying to launch the MongoDB blueprint (‘mongo-sharded.yaml’) but I’m
>>> facing the following error. The cluster keeps on resizing ('Resize
>>> MongoDBReplicaSetImpl{id=UGwzKMYx} from 0 to 3’) calling start()
>>> multiple times until it runs out of memory:
>>>
>>> Service is not up when setting running on MongoDBReplicaSetImpl{id=ABrSIbr2};
>>> delayed 1s 377ms but Sensor: service.isUp (java.lang.Boolean) did not
>>> recover from false; not-up-indicators={service-not-up-indicators-from-
>>> children-and-members=MongoDBReplicaSetImpl{id=vAqS7FvJ} is not up}
>>>
>>> 2014-11-03 14:41:18,656 WARN Attempt #1/1 to obtain machine threw error:
>>> com.google.inject.internal.util.$ComputationException:
>>> com.google.inject.internal.util.$ComputationException:
>>> java.lang.OutOfMemoryError: PermGen space
>>>
>>> 2014-11-03 14:41:18,657 WARN Failed to create entity using spec
>>> EntitySpec{type=interface brooklyn.entity.nosql.mongodb.MongoDBReplicaSet}
>>> (rethrowing)
>>> java.lang.OutOfMemoryError: PermGen space
>>>
>>> Anyone know what’s the solution here?
>>>
>>> Regards
>>>
>>
>
Re: MongoDBReplicaSetImpl error (out of memory)
Posted by Martin Harris <ma...@cloudsoftcorp.com>.
I think the real problem is that .start() is being called repeatedly. I
know there have been some changes to doStart() etc recently, so it may be
related to that
Cheers
M
On 3 November 2014 17:08, Aled Sage <al...@gmail.com> wrote:
> Hi Zaid,
>
> How are you running this? Is it with the `brooklyn` script or via an IDE?
>
> Depending on how much you are planning to run in the Brooklyn instance,
> you'll want to launch it with something like:
>
> -Xms512m -Xmx512m -XX:MaxPermSize=256m
>
> Running out of PermGen means that it has run out of memory for loading
> classes and for interned strings (i.e. constants). The last of the
> arguments above increases the space available.
>
> If you are running with the `brooklyn` script, it should already be doing
> this.
>
> ---
> Not sure I followed your description: it keeps resizing for 0 to 3
> multiple times? How many times has it tried? Is that with the same
> MongoDBReplicaSetImpl (id=UGwzKMYx) each time?
>
> Aled
>
>
>
> On 03/11/2014 14:44, Zaid Mohsin wrote:
>
>> Hi,
>> I’m trying to launch the MongoDB blueprint (‘mongo-sharded.yaml’) but I’m
>> facing the following error. The cluster keeps on resizing ('Resize
>> MongoDBReplicaSetImpl{id=UGwzKMYx} from 0 to 3’) calling start()
>> multiple times until it runs out of memory:
>>
>> Service is not up when setting running on MongoDBReplicaSetImpl{id=ABrSIbr2};
>> delayed 1s 377ms but Sensor: service.isUp (java.lang.Boolean) did not
>> recover from false; not-up-indicators={service-not-up-indicators-from-
>> children-and-members=MongoDBReplicaSetImpl{id=vAqS7FvJ} is not up}
>>
>> 2014-11-03 14:41:18,656 WARN Attempt #1/1 to obtain machine threw error:
>> com.google.inject.internal.util.$ComputationException:
>> com.google.inject.internal.util.$ComputationException:
>> java.lang.OutOfMemoryError: PermGen space
>>
>> 2014-11-03 14:41:18,657 WARN Failed to create entity using spec
>> EntitySpec{type=interface brooklyn.entity.nosql.mongodb.MongoDBReplicaSet}
>> (rethrowing)
>> java.lang.OutOfMemoryError: PermGen space
>>
>> Anyone know what’s the solution here?
>>
>> Regards
>>
>
>
--
Martin Harris
Lead Software Engineer
Cloudsoft Corporation Ltd
www.cloudsoftcorp.com
Mobile: +44 (0)7989 047-855
Re: MongoDBReplicaSetImpl error (out of memory)
Posted by Aled Sage <al...@gmail.com>.
Hi Zaid,
How are you running this? Is it with the `brooklyn` script or via an IDE?
Depending on how much you are planning to run in the Brooklyn instance,
you'll want to launch it with something like:
-Xms512m -Xmx512m -XX:MaxPermSize=256m
Running out of PermGen means that it has run out of memory for loading
classes and for interned strings (i.e. constants). The last of the
arguments above increases the space available.
If you are running with the `brooklyn` script, it should already be
doing this.
---
Not sure I followed your description: it keeps resizing for 0 to 3
multiple times? How many times has it tried? Is that with the same
MongoDBReplicaSetImpl (id=UGwzKMYx) each time?
Aled
On 03/11/2014 14:44, Zaid Mohsin wrote:
> Hi,
> I’m trying to launch the MongoDB blueprint (‘mongo-sharded.yaml’) but I’m facing the following error. The cluster keeps on resizing ('Resize MongoDBReplicaSetImpl{id=UGwzKMYx} from 0 to 3’) calling start() multiple times until it runs out of memory:
>
> Service is not up when setting running on MongoDBReplicaSetImpl{id=ABrSIbr2}; delayed 1s 377ms but Sensor: service.isUp (java.lang.Boolean) did not recover from false; not-up-indicators={service-not-up-indicators-from-children-and-members=MongoDBReplicaSetImpl{id=vAqS7FvJ} is not up}
>
> 2014-11-03 14:41:18,656 WARN Attempt #1/1 to obtain machine threw error: com.google.inject.internal.util.$ComputationException: com.google.inject.internal.util.$ComputationException: java.lang.OutOfMemoryError: PermGen space
>
> 2014-11-03 14:41:18,657 WARN Failed to create entity using spec EntitySpec{type=interface brooklyn.entity.nosql.mongodb.MongoDBReplicaSet} (rethrowing)
> java.lang.OutOfMemoryError: PermGen space
>
> Anyone know what’s the solution here?
>
> Regards