You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Daan Hoogland <da...@shapeblue.com> on 2017/04/11 07:30:02 UTC
growing #threads problem
Devs,
While I am creating a runnable tob e scheduled I ran into a problem. My scheduledJob nevers gets called and looking at the process the number of threads of type “DirectAgent-*” and “DirectAgentCronJob-*” keep growing. My job is scanning for templates in the primary storage and references to it in the database, so it seems to be unrelated. This is a 4.10 vmware based nested environment. Has anyone seen such a behaviour? I am still looking but I hope I won’t be reporting a blocker for 4.10 on this….
Any clue is welcome at this point,
daan.hoogland@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: growing #threads problem
Posted by Daan Hoogland <da...@shapeblue.com>.
I found the bug that let me to encounter this behaviour. I’ll keep an eye on this after I fixed it (simple stupid NPE). I don’t think it is a big problem as the DirectAgentCronJobs are maximised to 500 by default. Still strange that it keeps growing in a rather silent environment though.
On 11/04/17 13:11, "Daan Hoogland" <da...@shapeblue.com> wrote:
Nproc == -1 , both soft and hard so unless this is inherited from VMs in nested environments this can’t be the problem
Also no strange unexpected exceptions found, only failed connections to hosts during startup and environment configuration.
On 11/04/17 12:01, "Koushik Das" <ko...@accelerite.com> wrote:
Check the MS logs to see if there are OOM exceptions due to threads not available. If that’s the case then you may have to adjust nproc limits in /etc/security/limits.conf.
On 11/04/17, 1:11 PM, "Daan Hoogland" <da...@shapeblue.com> wrote:
My next step is to rebase my change to 4.9 and see if that works. But that is a stab in the dark I must admit.
On 11/04/17 09:30, "Daan Hoogland" <da...@shapeblue.com> wrote:
Devs,
While I am creating a runnable tob e scheduled I ran into a problem. My scheduledJob nevers gets called and looking at the process the number of threads of type “DirectAgent-*” and “DirectAgentCronJob-*” keep growing. My job is scanning for templates in the primary storage and references to it in the database, so it seems to be unrelated. This is a 4.10 vmware based nested environment. Has anyone seen such a behaviour? I am still looking but I hope I won’t be reporting a blocker for 4.10 on this….
Any clue is welcome at this point,
daan.hoogland@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
daan.hoogland@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
daan.hoogland@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
daan.hoogland@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: growing #threads problem
Posted by Daan Hoogland <da...@shapeblue.com>.
Nproc == -1 , both soft and hard so unless this is inherited from VMs in nested environments this can’t be the problem
Also no strange unexpected exceptions found, only failed connections to hosts during startup and environment configuration.
On 11/04/17 12:01, "Koushik Das" <ko...@accelerite.com> wrote:
Check the MS logs to see if there are OOM exceptions due to threads not available. If that’s the case then you may have to adjust nproc limits in /etc/security/limits.conf.
On 11/04/17, 1:11 PM, "Daan Hoogland" <da...@shapeblue.com> wrote:
My next step is to rebase my change to 4.9 and see if that works. But that is a stab in the dark I must admit.
On 11/04/17 09:30, "Daan Hoogland" <da...@shapeblue.com> wrote:
Devs,
While I am creating a runnable tob e scheduled I ran into a problem. My scheduledJob nevers gets called and looking at the process the number of threads of type “DirectAgent-*” and “DirectAgentCronJob-*” keep growing. My job is scanning for templates in the primary storage and references to it in the database, so it seems to be unrelated. This is a 4.10 vmware based nested environment. Has anyone seen such a behaviour? I am still looking but I hope I won’t be reporting a blocker for 4.10 on this….
Any clue is welcome at this point,
daan.hoogland@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
daan.hoogland@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
daan.hoogland@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: growing #threads problem
Posted by Koushik Das <ko...@accelerite.com>.
Check the MS logs to see if there are OOM exceptions due to threads not available. If that’s the case then you may have to adjust nproc limits in /etc/security/limits.conf.
On 11/04/17, 1:11 PM, "Daan Hoogland" <da...@shapeblue.com> wrote:
My next step is to rebase my change to 4.9 and see if that works. But that is a stab in the dark I must admit.
On 11/04/17 09:30, "Daan Hoogland" <da...@shapeblue.com> wrote:
Devs,
While I am creating a runnable tob e scheduled I ran into a problem. My scheduledJob nevers gets called and looking at the process the number of threads of type “DirectAgent-*” and “DirectAgentCronJob-*” keep growing. My job is scanning for templates in the primary storage and references to it in the database, so it seems to be unrelated. This is a 4.10 vmware based nested environment. Has anyone seen such a behaviour? I am still looking but I hope I won’t be reporting a blocker for 4.10 on this….
Any clue is welcome at this point,
daan.hoogland@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
daan.hoogland@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
Re: growing #threads problem
Posted by Daan Hoogland <da...@shapeblue.com>.
My next step is to rebase my change to 4.9 and see if that works. But that is a stab in the dark I must admit.
On 11/04/17 09:30, "Daan Hoogland" <da...@shapeblue.com> wrote:
Devs,
While I am creating a runnable tob e scheduled I ran into a problem. My scheduledJob nevers gets called and looking at the process the number of threads of type “DirectAgent-*” and “DirectAgentCronJob-*” keep growing. My job is scanning for templates in the primary storage and references to it in the database, so it seems to be unrelated. This is a 4.10 vmware based nested environment. Has anyone seen such a behaviour? I am still looking but I hope I won’t be reporting a blocker for 4.10 on this….
Any clue is welcome at this point,
daan.hoogland@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
daan.hoogland@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue