You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Ian Service <is...@ts2.ca> on 2014/11/04 20:33:27 UTC
Upgrade woes from CS 4.3 to 4.4.1
Updated system last night and now I'm getting the following error after
about 30 minutes of runtime:
java.lang.OutOfMemoryError: unable to create new native thread
At that point there's still 1.5GB of RAM in the management system and
there's very little load so I'm guessing I need to enable more threads to
make this work properly? Or is there something bigger here?
# uname -a
Linux man01 2.6.32-504.el6.x86_64 #1 SMP Wed Oct 15 04:27:16 UTC 2014
x86_64 x86_64 x86_64 GNU/Linux
# ps -L -u cloud | wc -l
1025
# cat /etc/security/limits.conf
cloud soft nofile 4096
cloud hard nofile 4096
Thanks for any help,
- Ian
Re: Upgrade woes from CS 4.3 to 4.4.1
Posted by Dave Dunaway <da...@gmail.com>.
check ulimit -u and/or /proc/sys/kernel/threads-max as well for potential
snafu's. As Pierre-Luc mentionned, it it said you should alllocate 8gigs of
Ramalamadingdongs on the mgmt box.
hth
On Tue, Nov 4, 2014 at 2:33 PM, Ian Service <is...@ts2.ca> wrote:
> Updated system last night and now I'm getting the following error after
> about 30 minutes of runtime:
>
> java.lang.OutOfMemoryError: unable to create new native thread
>
> At that point there's still 1.5GB of RAM in the management system and
> there's very little load so I'm guessing I need to enable more threads to
> make this work properly? Or is there something bigger here?
>
> # uname -a
> Linux man01 2.6.32-504.el6.x86_64 #1 SMP Wed Oct 15 04:27:16 UTC 2014
> x86_64 x86_64 x86_64 GNU/Linux
>
> # ps -L -u cloud | wc -l
> 1025
>
> # cat /etc/security/limits.conf
> cloud soft nofile 4096
> cloud hard nofile 4096
>
> Thanks for any help,
>
> - Ian
>
Re: Upgrade woes from CS 4.3 to 4.4.1
Posted by Ian Service <is...@ts2.ca>.
It's got 4GB + 4GB of swap.
But there's still lots of resources available. I've upped the number of
threads to 4096 to see if that helps.
Anyone running a functional cloudstack 4.4.1 that could check this number?
ps -L -u cloud | wc -l
- Ian
On Tue, Nov 4, 2014 at 2:54 PM, Pierre-Luc Dion <pd...@cloudops.com> wrote:
> how much memory does the management server have?
> Usually it worked best with 8GB.
>
>
>
> PL,
>
>
> On Tue, Nov 4, 2014 at 2:39 PM, Ian Service <is...@ts2.ca> wrote:
>
> > Also not sure if it has anything to do with it but getting a lot of these
> > messages on the console:
> >
> > Queue (queue id, sync type, sync id) - (811,VmWorkJobQueue, 28) is
> > reaching concurrency limit 1
> >
> > and
> >
> > vbdmetrics doesn't exist
> >
> > The latter appears to be from VMs that haven't been restarted yet, the
> line
> > includes vm cpu utilization on some machines.
> >
> > - Ian
> >
> > On Tue, Nov 4, 2014 at 2:33 PM, Ian Service <is...@ts2.ca> wrote:
> >
> > > Updated system last night and now I'm getting the following error after
> > > about 30 minutes of runtime:
> > >
> > > java.lang.OutOfMemoryError: unable to create new native thread
> > >
> > > At that point there's still 1.5GB of RAM in the management system and
> > > there's very little load so I'm guessing I need to enable more threads
> to
> > > make this work properly? Or is there something bigger here?
> > >
> > > # uname -a
> > > Linux man01 2.6.32-504.el6.x86_64 #1 SMP Wed Oct 15 04:27:16 UTC 2014
> > > x86_64 x86_64 x86_64 GNU/Linux
> > >
> > > # ps -L -u cloud | wc -l
> > > 1025
> > >
> > > # cat /etc/security/limits.conf
> > > cloud soft nofile 4096
> > > cloud hard nofile 4096
> > >
> > > Thanks for any help,
> > >
> > > - Ian
> > >
> >
>
Re: Upgrade woes from CS 4.3 to 4.4.1
Posted by Pierre-Luc Dion <pd...@cloudops.com>.
how much memory does the management server have?
Usually it worked best with 8GB.
PL,
On Tue, Nov 4, 2014 at 2:39 PM, Ian Service <is...@ts2.ca> wrote:
> Also not sure if it has anything to do with it but getting a lot of these
> messages on the console:
>
> Queue (queue id, sync type, sync id) - (811,VmWorkJobQueue, 28) is
> reaching concurrency limit 1
>
> and
>
> vbdmetrics doesn't exist
>
> The latter appears to be from VMs that haven't been restarted yet, the line
> includes vm cpu utilization on some machines.
>
> - Ian
>
> On Tue, Nov 4, 2014 at 2:33 PM, Ian Service <is...@ts2.ca> wrote:
>
> > Updated system last night and now I'm getting the following error after
> > about 30 minutes of runtime:
> >
> > java.lang.OutOfMemoryError: unable to create new native thread
> >
> > At that point there's still 1.5GB of RAM in the management system and
> > there's very little load so I'm guessing I need to enable more threads to
> > make this work properly? Or is there something bigger here?
> >
> > # uname -a
> > Linux man01 2.6.32-504.el6.x86_64 #1 SMP Wed Oct 15 04:27:16 UTC 2014
> > x86_64 x86_64 x86_64 GNU/Linux
> >
> > # ps -L -u cloud | wc -l
> > 1025
> >
> > # cat /etc/security/limits.conf
> > cloud soft nofile 4096
> > cloud hard nofile 4096
> >
> > Thanks for any help,
> >
> > - Ian
> >
>
Re: Upgrade woes from CS 4.3 to 4.4.1
Posted by Ian Service <is...@ts2.ca>.
Also not sure if it has anything to do with it but getting a lot of these
messages on the console:
Queue (queue id, sync type, sync id) - (811,VmWorkJobQueue, 28) is
reaching concurrency limit 1
and
vbdmetrics doesn't exist
The latter appears to be from VMs that haven't been restarted yet, the line
includes vm cpu utilization on some machines.
- Ian
On Tue, Nov 4, 2014 at 2:33 PM, Ian Service <is...@ts2.ca> wrote:
> Updated system last night and now I'm getting the following error after
> about 30 minutes of runtime:
>
> java.lang.OutOfMemoryError: unable to create new native thread
>
> At that point there's still 1.5GB of RAM in the management system and
> there's very little load so I'm guessing I need to enable more threads to
> make this work properly? Or is there something bigger here?
>
> # uname -a
> Linux man01 2.6.32-504.el6.x86_64 #1 SMP Wed Oct 15 04:27:16 UTC 2014
> x86_64 x86_64 x86_64 GNU/Linux
>
> # ps -L -u cloud | wc -l
> 1025
>
> # cat /etc/security/limits.conf
> cloud soft nofile 4096
> cloud hard nofile 4096
>
> Thanks for any help,
>
> - Ian
>