You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Ryan Dietrich <ry...@betterservers.com> on 2013/07/02 22:17:43 UTC
Async Job / Event Bus issue
I've found a bug that I am having trouble determining the cause. Basically, my instance_type and instance_id columns are showing up when the job is submitted, but they do not show up when the job is completed or updated.
When I look in the database, those columns are null, however, they are published correctly to the event bus on submit.
I went a step further and verified that SqlGenerator.java is actually using these columns on insertion and for querying the fields back out.
INSERT INTO async_job (async_job.user_id, async_job.account_id, async_job.session_key, async_job.job_cmd, async_job.job_cmd_originator, async_job.job_cmd_ver, async_job.job_cmd_info, async_job.callback_type, async_job.callback_address, async_job.job_status, async_job.job_process_status, async_job.job_re
sult_code, async_job.job_result, async_job.instance_type, async_job.instance_id, async_job.job_init_msid, async_job.job_complete_msid, async_job.created, async_job.last_updated, async_job.last_polled, async_job.uuid) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
I also verified that the column has a database @Column annotation. SqlGenerator appears to be using reflection to determine the "attributes" that get mapped into columns.
(from AsyncJobVO.java)
@Enumerated(value=EnumType.STRING)
@Column(name="instance_type", length=64)
private Type instanceType;
@Column(name="instance_id", length=64)
private Long instanceId;
What could be causing the instance_id and instance_type to drop on the floor? I am literally logging the job right after it says it has been written to the database, I log it again right after it is retrieved from the dao via findById. Either the dao isn't populating the volumes when it queries the AsyncJobVO back out (and subsequently updates the columns with null values), or it's something that I'm simply not seeing.
Help?
On Jul 2, 2013, at 8:15 AM, Ryan Dietrich <ry...@betterservers.com> wrote:
> Patch submitted.
>
> https://reviews.apache.org/r/12223/
>
> Things that still need to be thought through:
>
> 1. Should the async command event type be a column on async job (and be an attribute of AsyncJobVO)?
> 2. Should getInstanceId and getInstanceType of AsyncCmd be abstract instead of implemented with null values? (would require updating every async command in the system, and would break everyones async plugin commands)
>
> Oh, and after figuring out how the parameter annotation works (and ApiDispatcher uses reflection to iterate over parameters, ugh), I saw BaseCmd.java has a HashMap<String, String> of the arguments passed in from the user.
> I went with the APIDBUtils query function for UUID's because I saw no precedent for accessing that HashMap after initial argument handling.
>
> -Ryan Dietrich
RE: Async Job / Event Bus issue
Posted by Alex Huang <Al...@citrix.com>.
Ryan,
There's explicit code to remove those two columns. IIRC, the reason is because asyncjob actually uses those two fields to sequence jobs on the same id and type. So when the job is done, then it removes those two fields so it doesn't block the next job.
--Alex
> -----Original Message-----
> From: Ryan Dietrich [mailto:ryan@betterservers.com]
> Sent: Tuesday, July 2, 2013 1:18 PM
> To: dev@cloudstack.apache.org
> Subject: Async Job / Event Bus issue
>
> I've found a bug that I am having trouble determining the cause. Basically,
> my instance_type and instance_id columns are showing up when the job is
> submitted, but they do not show up when the job is completed or updated.
> When I look in the database, those columns are null, however, they are
> published correctly to the event bus on submit.
>
> I went a step further and verified that SqlGenerator.java is actually using
> these columns on insertion and for querying the fields back out.
>
> INSERT INTO async_job (async_job.user_id, async_job.account_id,
> async_job.session_key, async_job.job_cmd, async_job.job_cmd_originator,
> async_job.job_cmd_ver, async_job.job_cmd_info, async_job.callback_type,
> async_job.callback_address, async_job.job_status,
> async_job.job_process_status, async_job.job_re sult_code,
> async_job.job_result, async_job.instance_type, async_job.instance_id,
> async_job.job_init_msid, async_job.job_complete_msid, async_job.created,
> async_job.last_updated, async_job.last_polled, async_job.uuid) VALUES
> (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
>
> I also verified that the column has a database @Column annotation.
> SqlGenerator appears to be using reflection to determine the "attributes"
> that get mapped into columns.
>
> (from AsyncJobVO.java)
> @Enumerated(value=EnumType.STRING)
> @Column(name="instance_type", length=64)
> private Type instanceType;
>
> @Column(name="instance_id", length=64)
> private Long instanceId;
>
> What could be causing the instance_id and instance_type to drop on the
> floor? I am literally logging the job right after it says it has been written to the
> database, I log it again right after it is retrieved from the dao via findById.
> Either the dao isn't populating the volumes when it queries the AsyncJobVO
> back out (and subsequently updates the columns with null values), or it's
> something that I'm simply not seeing.
>
> Help?
>
> On Jul 2, 2013, at 8:15 AM, Ryan Dietrich <ry...@betterservers.com> wrote:
>
> > Patch submitted.
> >
> > https://reviews.apache.org/r/12223/
> >
> > Things that still need to be thought through:
> >
> > 1. Should the async command event type be a column on async job (and be
> an attribute of AsyncJobVO)?
> > 2. Should getInstanceId and getInstanceType of AsyncCmd be abstract
> > instead of implemented with null values? (would require updating every
> > async command in the system, and would break everyones async plugin
> > commands)
> >
> > Oh, and after figuring out how the parameter annotation works (and
> ApiDispatcher uses reflection to iterate over parameters, ugh), I saw
> BaseCmd.java has a HashMap<String, String> of the arguments passed in
> from the user.
> > I went with the APIDBUtils query function for UUID's because I saw no
> precedent for accessing that HashMap after initial argument handling.
> >
> > -Ryan Dietrich
Re: Async Job / Event Bus issue
Posted by Marcus Sorensen <sh...@gmail.com>.
He has a point though, why go to the trouble of removing the data to
tell that a job is no longer active, when there's a field that tells
us if the job is active? Was there some other reason?
On Tue, Jul 2, 2013 at 4:42 PM, Kelven Yang <ke...@citrix.com> wrote:
> Fields in job record are tend to be used to reflect job status in job's
> life cycle instead of history.
>
> Kelven
>
> On 7/2/13 1:35 PM, "Ryan Dietrich" <ry...@betterservers.com> wrote:
>
>>Ok. I see the "reset attached object" calls now.
>>
>>I'm curious why those columns are being wiped out. If you wanted to see
>>outstanding calls for an ID or Type, wouldn't you just query where the
>>jobStatus is still equal to zero?
>>
>>Wiping out those columns seems to destroy history, no?
>>
>>On Jul 2, 2013, at 2:27 PM, Alena Prokharchyk
>><Al...@citrix.com> wrote:
>>
>>> Ryan,
>>>
>>> Those fields were added to indicate the instances (volumes, vms, etc)
>>> having pending/active async jobs. The customers' main use case was - by
>>> instance id, find out the jobs that are currently being executed for
>>>this
>>> instance. Once the job is completed, the field is offset to NULL.
>>>
>>> -Alena.
>>
>
Re: Async Job / Event Bus issue
Posted by Murali Reddy <Mu...@citrix.com>.
What do we break if we fix it now? Since instance id, instance type are
also part of the 'AsyncJobResponse' its appropriate to pass full details.
Clearly from the Ryan use case, details of job are needed by external
systems and since this is the only book-keeping of a job, we need to
retain the history.
On 03/07/13 4:12 AM, "Kelven Yang" <ke...@citrix.com> wrote:
>Fields in job record are tend to be used to reflect job status in job's
>life cycle instead of history.
>
>Kelven
>
>On 7/2/13 1:35 PM, "Ryan Dietrich" <ry...@betterservers.com> wrote:
>
>>Ok. I see the "reset attached object" calls now.
>>
>>I'm curious why those columns are being wiped out. If you wanted to see
>>outstanding calls for an ID or Type, wouldn't you just query where the
>>jobStatus is still equal to zero?
>>
>>Wiping out those columns seems to destroy history, no?
>>
>>On Jul 2, 2013, at 2:27 PM, Alena Prokharchyk
>><Al...@citrix.com> wrote:
>>
>>> Ryan,
>>>
>>> Those fields were added to indicate the instances (volumes, vms, etc)
>>> having pending/active async jobs. The customers' main use case was - by
>>> instance id, find out the jobs that are currently being executed for
>>>this
>>> instance. Once the job is completed, the field is offset to NULL.
>>>
>>> -Alena.
>>
>
>
Re: Async Job / Event Bus issue
Posted by Kelven Yang <ke...@citrix.com>.
Fields in job record are tend to be used to reflect job status in job's
life cycle instead of history.
Kelven
On 7/2/13 1:35 PM, "Ryan Dietrich" <ry...@betterservers.com> wrote:
>Ok. I see the "reset attached object" calls now.
>
>I'm curious why those columns are being wiped out. If you wanted to see
>outstanding calls for an ID or Type, wouldn't you just query where the
>jobStatus is still equal to zero?
>
>Wiping out those columns seems to destroy history, no?
>
>On Jul 2, 2013, at 2:27 PM, Alena Prokharchyk
><Al...@citrix.com> wrote:
>
>> Ryan,
>>
>> Those fields were added to indicate the instances (volumes, vms, etc)
>> having pending/active async jobs. The customers' main use case was - by
>> instance id, find out the jobs that are currently being executed for
>>this
>> instance. Once the job is completed, the field is offset to NULL.
>>
>> -Alena.
>
Re: Async Job / Event Bus issue
Posted by Ryan Dietrich <ry...@betterservers.com>.
Ok. I see the "reset attached object" calls now.
I'm curious why those columns are being wiped out. If you wanted to see outstanding calls for an ID or Type, wouldn't you just query where the jobStatus is still equal to zero?
Wiping out those columns seems to destroy history, no?
On Jul 2, 2013, at 2:27 PM, Alena Prokharchyk <Al...@citrix.com> wrote:
> Ryan,
>
> Those fields were added to indicate the instances (volumes, vms, etc)
> having pending/active async jobs. The customers' main use case was - by
> instance id, find out the jobs that are currently being executed for this
> instance. Once the job is completed, the field is offset to NULL.
>
> -Alena.
Re: Async Job / Event Bus issue
Posted by Alena Prokharchyk <Al...@citrix.com>.
Ryan,
Those fields were added to indicate the instances (volumes, vms, etc)
having pending/active async jobs. The customers' main use case was - by
instance id, find out the jobs that are currently being executed for this
instance. Once the job is completed, the field is offset to NULL.
-Alena.
On 7/2/13 1:17 PM, "Ryan Dietrich" <ry...@betterservers.com> wrote:
>I've found a bug that I am having trouble determining the cause.
>Basically, my instance_type and instance_id columns are showing up when
>the job is submitted, but they do not show up when the job is completed
>or updated.
>When I look in the database, those columns are null, however, they are
>published correctly to the event bus on submit.
>
>I went a step further and verified that SqlGenerator.java is actually
>using these columns on insertion and for querying the fields back out.
>
>INSERT INTO async_job (async_job.user_id, async_job.account_id,
>async_job.session_key, async_job.job_cmd, async_job.job_cmd_originator,
>async_job.job_cmd_ver, async_job.job_cmd_info, async_job.callback_type,
>async_job.callback_address, async_job.job_status,
>async_job.job_process_status, async_job.job_re
>sult_code, async_job.job_result, async_job.instance_type,
>async_job.instance_id, async_job.job_init_msid,
>async_job.job_complete_msid, async_job.created, async_job.last_updated,
>async_job.last_polled, async_job.uuid) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?,
>?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
>
>I also verified that the column has a database @Column annotation.
>SqlGenerator appears to be using reflection to determine the "attributes"
>that get mapped into columns.
>
>(from AsyncJobVO.java)
> @Enumerated(value=EnumType.STRING)
> @Column(name="instance_type", length=64)
> private Type instanceType;
>
> @Column(name="instance_id", length=64)
> private Long instanceId;
>
>What could be causing the instance_id and instance_type to drop on the
>floor? I am literally logging the job right after it says it has been
>written to the database, I log it again right after it is retrieved from
>the dao via findById. Either the dao isn't populating the volumes when
>it queries the AsyncJobVO back out (and subsequently updates the columns
>with null values), or it's something that I'm simply not seeing.
>
>Help?
>
>On Jul 2, 2013, at 8:15 AM, Ryan Dietrich <ry...@betterservers.com> wrote:
>
>> Patch submitted.
>>
>> https://reviews.apache.org/r/12223/
>>
>> Things that still need to be thought through:
>>
>> 1. Should the async command event type be a column on async job (and be
>>an attribute of AsyncJobVO)?
>> 2. Should getInstanceId and getInstanceType of AsyncCmd be abstract
>>instead of implemented with null values? (would require updating every
>>async command in the system, and would break everyones async plugin
>>commands)
>>
>> Oh, and after figuring out how the parameter annotation works (and
>>ApiDispatcher uses reflection to iterate over parameters, ugh), I saw
>>BaseCmd.java has a HashMap<String, String> of the arguments passed in
>>from the user.
>> I went with the APIDBUtils query function for UUID's because I saw no
>>precedent for accessing that HashMap after initial argument handling.
>>
>> -Ryan Dietrich
>
>