You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Francois Gaudreault <fg...@cloudops.com> on 2015/01/05 22:40:58 UTC
Cancel hung jobs
Hi folks,
What would be the best way to cancel jobs that are "stuck"? I was
expecting them to timeout, but for some reasons they are not.
Thanks.
--
Francois Gaudreault
Gestionnaire de Produit | Product Manager - Cloud Platform & Services
t:514-629-6775
CloudOps Votre partenaire infonuagique | Cloud Solutions Experts
420 rue Guy | Montreal | Quebec | H3J 1S6
w: cloudops.com | tw: @CloudOps_
Re: Cancel hung jobs
Posted by Francois Gaudreault <fg...@cloudops.com>.
Thanks for the info.
I finally ended up restarting the management servers since there was a
DB split-brain issue underneath.
FG
On 2015-01-06 12:32 PM, ilya musayev wrote:
> Here are my notes on cleaning up stuck VmWorkJobQueue, i use MySQL
> workbench to make changes interactively...
>
> Messages in log file:
>
> (1013,VmWorkJobQueue, 2375) is reaching concurrency limit 1
> (1013,VmWorkJobQueue, 2375) is reaching concurrency limit 1
> (1433,VmWorkJobQueue, 2470) is reaching concurrency limit 1
> (1434,VmWorkJobQueue, 2471) is reaching concurrency limit 1
> (1475,VmWorkJobQueue, 2483) is reaching concurrency limit 1
> (1477,VmWorkJobQueue, 2484) is reaching concurrency limit 1
> (1478,VmWorkJobQueue, 596) is reaching concurrency limit 1
> (981,VmWorkJobQueue, 2371) is reaching concurrency limit 1
>
> ------
>
>
> DELETE ALL FROM BELOW - not exact mysql query, i just did select first
> and then deleted rows via mysql workbench.....
>
> SELECT * FROM cloud.sync_queue_item where queue_id=981 or
> queue_id=1478 or queue_id=1477 or queue_id=1475 or queue_id=1434 or
> queue_id=1433 or queue_id=1013 ;
>
> SELECT * FROM cloud.sync_queue where id=1433 or id=1434 or id=1475 or
> id=1477 or id=1478 or id=981;
>
> On 1/6/15 1:27 AM, Prashant Kumar Mishra wrote:
>> Restart MS should work . If you don¹t want to restart clean it manually
>> from db , it seems because of one unfinished job , other jobs is also
>> pending .
>>
>> On 1/6/15, 4:09 AM, "Francois Gaudreault" <fg...@cloudops.com>
>> wrote:
>>
>>> These are 60min and 1440min for standard jobs and 600sec for vm related
>>> jobs right? I have tasks that are from dec 27 still in there.... and
>>> these
>>> are vm jobs.
>>>
>>> FG
>>> On Jan 5, 2015 5:25 PM, "Motty Cruz" <mo...@gmail.com> wrote:
>>>
>>>> Francois,
>>>> you can change timeout default settings in "global settings", if you
>>>> have
>>>> not done so.
>>>>
>>>> Thanks,
>>>> Motty
>>>> On 01/05/2015 02:09 PM, Francois Gaudreault wrote:
>>>>
>>>>> Yes. And also tons of job in the async_job table with status 0.
>>>>>
>>>>> FG
>>>>>
>>>>> On 2015-01-05 4:54 PM, ilya musayev wrote:
>>>>>
>>>>>> Do you see something along the lines of below? please provide the
>>>>>> complete error message.
>>>>>>
>>>>>> (1434,VmWorkJobQueue, 2471) is reaching concurrency limit 1
>>>>>> (1475,VmWorkJobQueue, 2483) is reaching concurrency limit 1
>>>>>> (1477,VmWorkJobQueue, 2484) is reaching concurrency limit 1
>>>>>> (1478,VmWorkJobQueue, 596) is reaching concurrency limit 1
>>>>>> (981,VmWorkJobQueue, 2371) is reaching concurrency limit 1
>>>>>>
>>>>>>
>>>>>> On 1/5/15, 1:40 PM, Francois Gaudreault wrote:
>>>>>>
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> What would be the best way to cancel jobs that are "stuck"? I was
>>>>>>> expecting them to timeout, but for some reasons they are not.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>
>
>
--
Francois Gaudreault
Gestionnaire de Produit | Product Manager - Cloud Platform & Services
t:514-629-6775
CloudOps Votre partenaire infonuagique | Cloud Solutions Experts
420 rue Guy | Montreal | Quebec | H3J 1S6
w: cloudops.com | tw: @CloudOps_
Re: Cancel hung jobs
Posted by ilya musayev <il...@gmail.com>.
Here are my notes on cleaning up stuck VmWorkJobQueue, i use MySQL
workbench to make changes interactively...
Messages in log file:
(1013,VmWorkJobQueue, 2375) is reaching concurrency limit 1
(1013,VmWorkJobQueue, 2375) is reaching concurrency limit 1
(1433,VmWorkJobQueue, 2470) is reaching concurrency limit 1
(1434,VmWorkJobQueue, 2471) is reaching concurrency limit 1
(1475,VmWorkJobQueue, 2483) is reaching concurrency limit 1
(1477,VmWorkJobQueue, 2484) is reaching concurrency limit 1
(1478,VmWorkJobQueue, 596) is reaching concurrency limit 1
(981,VmWorkJobQueue, 2371) is reaching concurrency limit 1
------
DELETE ALL FROM BELOW - not exact mysql query, i just did select first and then deleted rows via mysql workbench.....
SELECT * FROM cloud.sync_queue_item where queue_id=981 or queue_id=1478 or queue_id=1477 or queue_id=1475 or queue_id=1434 or queue_id=1433 or queue_id=1013 ;
SELECT * FROM cloud.sync_queue where id=1433 or id=1434 or id=1475 or id=1477 or id=1478 or id=981;
On 1/6/15 1:27 AM, Prashant Kumar Mishra wrote:
> Restart MS should work . If you don¹t want to restart clean it manually
> from db , it seems because of one unfinished job , other jobs is also
> pending .
>
> On 1/6/15, 4:09 AM, "Francois Gaudreault" <fg...@cloudops.com> wrote:
>
>> These are 60min and 1440min for standard jobs and 600sec for vm related
>> jobs right? I have tasks that are from dec 27 still in there.... and these
>> are vm jobs.
>>
>> FG
>> On Jan 5, 2015 5:25 PM, "Motty Cruz" <mo...@gmail.com> wrote:
>>
>>> Francois,
>>> you can change timeout default settings in "global settings", if you
>>> have
>>> not done so.
>>>
>>> Thanks,
>>> Motty
>>> On 01/05/2015 02:09 PM, Francois Gaudreault wrote:
>>>
>>>> Yes. And also tons of job in the async_job table with status 0.
>>>>
>>>> FG
>>>>
>>>> On 2015-01-05 4:54 PM, ilya musayev wrote:
>>>>
>>>>> Do you see something along the lines of below? please provide the
>>>>> complete error message.
>>>>>
>>>>> (1434,VmWorkJobQueue, 2471) is reaching concurrency limit 1
>>>>> (1475,VmWorkJobQueue, 2483) is reaching concurrency limit 1
>>>>> (1477,VmWorkJobQueue, 2484) is reaching concurrency limit 1
>>>>> (1478,VmWorkJobQueue, 596) is reaching concurrency limit 1
>>>>> (981,VmWorkJobQueue, 2371) is reaching concurrency limit 1
>>>>>
>>>>>
>>>>> On 1/5/15, 1:40 PM, Francois Gaudreault wrote:
>>>>>
>>>>>> Hi folks,
>>>>>>
>>>>>> What would be the best way to cancel jobs that are "stuck"? I was
>>>>>> expecting them to timeout, but for some reasons they are not.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
Re: Cancel hung jobs
Posted by Prashant Kumar Mishra <pr...@citrix.com>.
Restart MS should work . If you don¹t want to restart clean it manually
from db , it seems because of one unfinished job , other jobs is also
pending .
On 1/6/15, 4:09 AM, "Francois Gaudreault" <fg...@cloudops.com> wrote:
>These are 60min and 1440min for standard jobs and 600sec for vm related
>jobs right? I have tasks that are from dec 27 still in there.... and these
>are vm jobs.
>
>FG
>On Jan 5, 2015 5:25 PM, "Motty Cruz" <mo...@gmail.com> wrote:
>
>> Francois,
>> you can change timeout default settings in "global settings", if you
>>have
>> not done so.
>>
>> Thanks,
>> Motty
>> On 01/05/2015 02:09 PM, Francois Gaudreault wrote:
>>
>>> Yes. And also tons of job in the async_job table with status 0.
>>>
>>> FG
>>>
>>> On 2015-01-05 4:54 PM, ilya musayev wrote:
>>>
>>>> Do you see something along the lines of below? please provide the
>>>> complete error message.
>>>>
>>>> (1434,VmWorkJobQueue, 2471) is reaching concurrency limit 1
>>>> (1475,VmWorkJobQueue, 2483) is reaching concurrency limit 1
>>>> (1477,VmWorkJobQueue, 2484) is reaching concurrency limit 1
>>>> (1478,VmWorkJobQueue, 596) is reaching concurrency limit 1
>>>> (981,VmWorkJobQueue, 2371) is reaching concurrency limit 1
>>>>
>>>>
>>>> On 1/5/15, 1:40 PM, Francois Gaudreault wrote:
>>>>
>>>>> Hi folks,
>>>>>
>>>>> What would be the best way to cancel jobs that are "stuck"? I was
>>>>> expecting them to timeout, but for some reasons they are not.
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
Re: Cancel hung jobs
Posted by Francois Gaudreault <fg...@cloudops.com>.
These are 60min and 1440min for standard jobs and 600sec for vm related
jobs right? I have tasks that are from dec 27 still in there.... and these
are vm jobs.
FG
On Jan 5, 2015 5:25 PM, "Motty Cruz" <mo...@gmail.com> wrote:
> Francois,
> you can change timeout default settings in "global settings", if you have
> not done so.
>
> Thanks,
> Motty
> On 01/05/2015 02:09 PM, Francois Gaudreault wrote:
>
>> Yes. And also tons of job in the async_job table with status 0.
>>
>> FG
>>
>> On 2015-01-05 4:54 PM, ilya musayev wrote:
>>
>>> Do you see something along the lines of below? please provide the
>>> complete error message.
>>>
>>> (1434,VmWorkJobQueue, 2471) is reaching concurrency limit 1
>>> (1475,VmWorkJobQueue, 2483) is reaching concurrency limit 1
>>> (1477,VmWorkJobQueue, 2484) is reaching concurrency limit 1
>>> (1478,VmWorkJobQueue, 596) is reaching concurrency limit 1
>>> (981,VmWorkJobQueue, 2371) is reaching concurrency limit 1
>>>
>>>
>>> On 1/5/15, 1:40 PM, Francois Gaudreault wrote:
>>>
>>>> Hi folks,
>>>>
>>>> What would be the best way to cancel jobs that are "stuck"? I was
>>>> expecting them to timeout, but for some reasons they are not.
>>>>
>>>> Thanks.
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>
Re: Cancel hung jobs
Posted by Motty Cruz <mo...@gmail.com>.
Francois,
you can change timeout default settings in "global settings", if you
have not done so.
Thanks,
Motty
On 01/05/2015 02:09 PM, Francois Gaudreault wrote:
> Yes. And also tons of job in the async_job table with status 0.
>
> FG
>
> On 2015-01-05 4:54 PM, ilya musayev wrote:
>> Do you see something along the lines of below? please provide the
>> complete error message.
>>
>> (1434,VmWorkJobQueue, 2471) is reaching concurrency limit 1
>> (1475,VmWorkJobQueue, 2483) is reaching concurrency limit 1
>> (1477,VmWorkJobQueue, 2484) is reaching concurrency limit 1
>> (1478,VmWorkJobQueue, 596) is reaching concurrency limit 1
>> (981,VmWorkJobQueue, 2371) is reaching concurrency limit 1
>>
>>
>> On 1/5/15, 1:40 PM, Francois Gaudreault wrote:
>>> Hi folks,
>>>
>>> What would be the best way to cancel jobs that are "stuck"? I was
>>> expecting them to timeout, but for some reasons they are not.
>>>
>>> Thanks.
>>>
>>
>>
>>
>
>
Re: Cancel hung jobs
Posted by Francois Gaudreault <fg...@cloudops.com>.
Yes. And also tons of job in the async_job table with status 0.
FG
On 2015-01-05 4:54 PM, ilya musayev wrote:
> Do you see something along the lines of below? please provide the
> complete error message.
>
> (1434,VmWorkJobQueue, 2471) is reaching concurrency limit 1
> (1475,VmWorkJobQueue, 2483) is reaching concurrency limit 1
> (1477,VmWorkJobQueue, 2484) is reaching concurrency limit 1
> (1478,VmWorkJobQueue, 596) is reaching concurrency limit 1
> (981,VmWorkJobQueue, 2371) is reaching concurrency limit 1
>
>
> On 1/5/15, 1:40 PM, Francois Gaudreault wrote:
>> Hi folks,
>>
>> What would be the best way to cancel jobs that are "stuck"? I was
>> expecting them to timeout, but for some reasons they are not.
>>
>> Thanks.
>>
>
>
>
--
Francois Gaudreault
Gestionnaire de Produit | Product Manager - Cloud Platform & Services
t:514-629-6775
CloudOps Votre partenaire infonuagique | Cloud Solutions Experts
420 rue Guy | Montreal | Quebec | H3J 1S6
w: cloudops.com | tw: @CloudOps_
Re: Cancel hung jobs
Posted by ilya musayev <il...@gmail.com>.
Do you see something along the lines of below? please provide the
complete error message.
(1434,VmWorkJobQueue, 2471) is reaching concurrency limit 1
(1475,VmWorkJobQueue, 2483) is reaching concurrency limit 1
(1477,VmWorkJobQueue, 2484) is reaching concurrency limit 1
(1478,VmWorkJobQueue, 596) is reaching concurrency limit 1
(981,VmWorkJobQueue, 2371) is reaching concurrency limit 1
On 1/5/15, 1:40 PM, Francois Gaudreault wrote:
> Hi folks,
>
> What would be the best way to cancel jobs that are "stuck"? I was
> expecting them to timeout, but for some reasons they are not.
>
> Thanks.
>