You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@deltacloud.apache.org by David Lutterkort <lu...@redhat.com> on 2012/06/08 12:31:32 UTC
Re: Deltacloud Tracker Design (Draft)
[ Adding dev@deltacloud ]
On Thu, 2012-06-07 at 21:15 +0200, Jan Provazník wrote:
> Hi Lutter,
> Michal pointed me to your draft of Tracker design which looks great.
> Couple of notes are inline. Sending it to aeolus-devel list too so
> others can join.
>
> Original doc can be found here:
> https://raw.github.com/lutter/deltacloud/tracker/design/site/content/tracker.mdown
And the pretty version is at
http://people.apache.org/~lutter/deltacloud/tracker.html
> > The resource changes that will trigger a notification to clients are
> > specific to each resource (and driver ?) They are
> >
> > * Instances: change to instance state
> > * [TODO: what else ?]
>
> Realms are needed too. We check realms availability in Conductor, so we
> will not be able to replace our current checking tool with Tracker until
> realms are supported too.
Realms is something that gets measured indirectly, right ? I.e., realm
availability is based on success/failure of launching/running instances
in that realm.
> > and to delete a callback, use the special token 'none':
> >
> > PUT /instances/42?track_hook=none
> >
>
> Authentication will be needed for ^ these requests to make sure that 3rd
> party doesn't modifies my callback.
The tacit assumption in the whole writeup is that all operations require
the backend auth credentials, like other DC requests; so to
delete/modify a hook, you'd have to prove that you can modify the
underlying resource.
> > To make it possible for recipients to catch up after a failure on their
> > side, we'll support a 'changes' collection that only allows GET:
> >
> > GET /changes
> >
> > The response to this request will be a JSON array, where each entry is the
> > same JSON object that is used for delivering callbacks. Note that only
> > changes pertinent to the current provider will be delivered, i.e. clients
> > that use Tracker to track resources in multiple providers will need to make
> > one request for each provider. Once a change has been delivered through
> > this mechanism, it will be considered successfully delivered.
> >
>
> This might be a problem if Tracker is used by multiple clients - _all_
> callbacks for a provider are fetched by one request no matter whom these
> callbacks are addressed to.
>
> Maybe 'GET /changes' could just trigger common retry of callback
> delivery for all provider requests instead of returning changes
> directly. Another benefit is that then 'GET /changes' wouldn't have to
> be authenticated.
Same as above: the assumption is that they are implicitly scoped by
backend account.
> > # Implementation Notes
> >
> > We will need to run a background job that performs the state
> > polling. DelayedJob seems like the right tool for this; we'll want a
>
> +1 for delayed job - we already have this requirement in conductor
What we need is slightly different from what delayed job gives you: we
need cron-like functionality to run tasks at fixed intervals, whereas
delayed job is used for handling one-shot tasks in the background. It's
possible to use delayed job for this, but hopefully somebody else has
come up with something that is closer. (Haven't looked yet)
David
Re: Deltacloud Tracker Design (Draft)
Posted by Michal Fojtik <mf...@redhat.com>.
On 06/08/12, David Lutterkort wrote:
> > Original doc can be found here:
> > https://raw.github.com/lutter/deltacloud/tracker/design/site/content/tracker.mdown
>
> And the pretty version is at
> http://people.apache.org/~lutter/deltacloud/tracker.html
And even prettier and publicly editable version can be found at:
http://openetherpad.org/DeltacloudTrackerDesign
-- Michal
-- Michal
--
Michal Fojtik
Sr. Software Engineer, Deltacloud API (http://deltacloud.org)
Re: Deltacloud Tracker Design (Draft)
Posted by Richard Su <rw...@redhat.com>.
On 06/08/2012 05:31 AM, David Lutterkort wrote:
> On Fri, 2012-06-08 at 13:43 +0200, Jan Provazník wrote:
>> On 06/08/2012 01:12 PM, Jan Provazník wrote:
>>> True, we were choosing some tool for this purpose on Aeolus side too few
>>> weeks/months ago (before we knew about Tracker). Richard Su made lot of
>>> work in this area (CCing him) so it would worh discuss this with him,
>>> wiki page with related stuff is here:
>> or even better his patches could be reused in Tracker
> Where can I find them ?
>
> David
>
Here's the patch to add delayed_job support to conductor:
https://fedorahosted.org/pipermail/aeolus-devel/2012-May/010236.html
That patch contains a systemd config file that starts up delayed_job as
a daemon with two workers. Not everyone has cut over to systemd, so
here's a patch containing a SysV init script:
https://fedorahosted.org/pipermail/aeolus-devel/2012-June/010771.html
The plan was then to use rufus-scheduler to periodically add jobs to the
queue to poll deltacloud for state changes. That plan has been subsumed
by the Deltacloud Tracker. I do have a patch that adds rufus-scheduler
support to conductor. Integrating the scheduler is not that difficult,
most of the code was adding some infrastructure to allow us to manage
the scheduler so that we can check its status and turn it on and off.
https://fedorahosted.org/pipermail/aeolus-devel/2012-May/010625.html
https://fedorahosted.org/pipermail/aeolus-devel/2012-May/010626.html
If you think the management piece may be of use, I can look into
rewriting it as a rack mountable app. That was a prescient suggestion by
Jay.
Re: Deltacloud Tracker Design (Draft)
Posted by David Lutterkort <lu...@redhat.com>.
On Fri, 2012-06-08 at 13:43 +0200, Jan Provazník wrote:
> On 06/08/2012 01:12 PM, Jan Provazník wrote:
> > True, we were choosing some tool for this purpose on Aeolus side too few
> > weeks/months ago (before we knew about Tracker). Richard Su made lot of
> > work in this area (CCing him) so it would worh discuss this with him,
> > wiki page with related stuff is here:
>
> or even better his patches could be reused in Tracker
Where can I find them ?
David
Re: Deltacloud Tracker Design (Draft)
Posted by Jan Provazník <jp...@redhat.com>.
On 06/08/2012 01:12 PM, Jan Provazník wrote:
>>>> # Implementation Notes
>>>>
>>>> We will need to run a background job that performs the state
>>>> polling. DelayedJob seems like the right tool for this; we'll want a
>>>
>>> +1 for delayed job - we already have this requirement in conductor
>>
>> What we need is slightly different from what delayed job gives you: we
>> need cron-like functionality to run tasks at fixed intervals, whereas
>> delayed job is used for handling one-shot tasks in the background. It's
>> possible to use delayed job for this, but hopefully somebody else has
>> come up with something that is closer. (Haven't looked yet)
>>
>
> True, we were choosing some tool for this purpose on Aeolus side too few
> weeks/months ago (before we knew about Tracker). Richard Su made lot of
> work in this area (CCing him) so it would worh discuss this with him,
> wiki page with related stuff is here:
or even better his patches could be reused in Tracker
> https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Background_Processing
>
>
> mail thread is here:
> https://fedorahosted.org/pipermail/aeolus-devel/2012-April/009923.html
>
>
Re: Deltacloud Tracker Design (Draft)
Posted by Jan Provaznik <jp...@redhat.com>.
On 06/08/2012 02:30 PM, David Lutterkort wrote:
>>>>> The resource changes that will trigger a notification to clients are
>>>>> specific to each resource (and driver ?) They are
>>>>>
>>>>> * Instances: change to instance state
>>>>> * [TODO: what else ?]
>>>>
>>>> Realms are needed too. We check realms availability in Conductor, so we
>>>> will not be able to replace our current checking tool with Tracker until
>>>> realms are supported too.
>>>
>>> Realms is something that gets measured indirectly, right ? I.e., realm
>>> availability is based on success/failure of launching/running instances
>>> in that realm.
>>>
>>
>> No, we check periodically realm's availability (unavailable realms are
>> then ignored when launching an instance).
>
> How do you check ?
>
Checking is now done by a ruby script
(https://github.com/aeolusproject/conductor/blob/master/src/dbomatic/dbomatic)
which fetches realms for each provider account every 5 minutes. The list
of fetched realms is then compared with realms in Conductor's DB -
availability status is updated, realms missing in DB are added, realms
which are in DB but not on the fetched list are removed.
We keep availability flag also for whole providers: if fetching of
realms for a provider account fails because of a connection error, whole
provider is marked as unavailable and skipped on instance launch.
Jan
Re: Deltacloud Tracker Design (Draft)
Posted by David Lutterkort <lu...@redhat.com>.
On Fri, 2012-06-08 at 13:12 +0200, Jan Provazník wrote:
> On 06/08/2012 12:31 PM, David Lutterkort wrote:
> > [ Adding dev@deltacloud ]
> >
> > On Thu, 2012-06-07 at 21:15 +0200, Jan Provazník wrote:
> >> Hi Lutter,
> >> Michal pointed me to your draft of Tracker design which looks great.
> >> Couple of notes are inline. Sending it to aeolus-devel list too so
> >> others can join.
> >>
> >> Original doc can be found here:
> >> https://raw.github.com/lutter/deltacloud/tracker/design/site/content/tracker.mdown
> >
> > And the pretty version is at
> > http://people.apache.org/~lutter/deltacloud/tracker.html
> >
> >>> The resource changes that will trigger a notification to clients are
> >>> specific to each resource (and driver ?) They are
> >>>
> >>> * Instances: change to instance state
> >>> * [TODO: what else ?]
> >>
> >> Realms are needed too. We check realms availability in Conductor, so we
> >> will not be able to replace our current checking tool with Tracker until
> >> realms are supported too.
> >
> > Realms is something that gets measured indirectly, right ? I.e., realm
> > availability is based on success/failure of launching/running instances
> > in that realm.
> >
>
> No, we check periodically realm's availability (unavailable realms are
> then ignored when launching an instance).
How do you check ?
> > What we need is slightly different from what delayed job gives you: we
> > need cron-like functionality to run tasks at fixed intervals, whereas
> > delayed job is used for handling one-shot tasks in the background. It's
> > possible to use delayed job for this, but hopefully somebody else has
> > come up with something that is closer. (Haven't looked yet)
> >
>
> True, we were choosing some tool for this purpose on Aeolus side too few
> weeks/months ago (before we knew about Tracker). Richard Su made lot of
> work in this area (CCing him) so it would worh discuss this with him,
> wiki page with related stuff is here:
> https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Background_Processing
>
> mail thread is here:
> https://fedorahosted.org/pipermail/aeolus-devel/2012-April/009923.html
Nice. Has anybody looked at clockwork[1] together with delayed job ? I
like the idea of separating the scheduler from the queue manager.
David
[1] https://github.com/adamwiggins/clockwork
Re: Deltacloud Tracker Design (Draft)
Posted by Jan Provazník <jp...@redhat.com>.
On 06/08/2012 12:31 PM, David Lutterkort wrote:
> [ Adding dev@deltacloud ]
>
> On Thu, 2012-06-07 at 21:15 +0200, Jan Provazník wrote:
>> Hi Lutter,
>> Michal pointed me to your draft of Tracker design which looks great.
>> Couple of notes are inline. Sending it to aeolus-devel list too so
>> others can join.
>>
>> Original doc can be found here:
>> https://raw.github.com/lutter/deltacloud/tracker/design/site/content/tracker.mdown
>
> And the pretty version is at
> http://people.apache.org/~lutter/deltacloud/tracker.html
>
>>> The resource changes that will trigger a notification to clients are
>>> specific to each resource (and driver ?) They are
>>>
>>> * Instances: change to instance state
>>> * [TODO: what else ?]
>>
>> Realms are needed too. We check realms availability in Conductor, so we
>> will not be able to replace our current checking tool with Tracker until
>> realms are supported too.
>
> Realms is something that gets measured indirectly, right ? I.e., realm
> availability is based on success/failure of launching/running instances
> in that realm.
>
No, we check periodically realm's availability (unavailable realms are
then ignored when launching an instance).
>>> and to delete a callback, use the special token 'none':
>>>
>>> PUT /instances/42?track_hook=none
>>>
>>
>> Authentication will be needed for ^ these requests to make sure that 3rd
>> party doesn't modifies my callback.
>
> The tacit assumption in the whole writeup is that all operations require
> the backend auth credentials, like other DC requests; so to
> delete/modify a hook, you'd have to prove that you can modify the
> underlying resource.
>
>>> To make it possible for recipients to catch up after a failure on their
>>> side, we'll support a 'changes' collection that only allows GET:
>>>
>>> GET /changes
>>>
>>> The response to this request will be a JSON array, where each entry is the
>>> same JSON object that is used for delivering callbacks. Note that only
>>> changes pertinent to the current provider will be delivered, i.e. clients
>>> that use Tracker to track resources in multiple providers will need to make
>>> one request for each provider. Once a change has been delivered through
>>> this mechanism, it will be considered successfully delivered.
>>>
>>
>> This might be a problem if Tracker is used by multiple clients - _all_
>> callbacks for a provider are fetched by one request no matter whom these
>> callbacks are addressed to.
>>
>> Maybe 'GET /changes' could just trigger common retry of callback
>> delivery for all provider requests instead of returning changes
>> directly. Another benefit is that then 'GET /changes' wouldn't have to
>> be authenticated.
>
> Same as above: the assumption is that they are implicitly scoped by
> backend account.
>
>>> # Implementation Notes
>>>
>>> We will need to run a background job that performs the state
>>> polling. DelayedJob seems like the right tool for this; we'll want a
>>
>> +1 for delayed job - we already have this requirement in conductor
>
> What we need is slightly different from what delayed job gives you: we
> need cron-like functionality to run tasks at fixed intervals, whereas
> delayed job is used for handling one-shot tasks in the background. It's
> possible to use delayed job for this, but hopefully somebody else has
> come up with something that is closer. (Haven't looked yet)
>
True, we were choosing some tool for this purpose on Aeolus side too few
weeks/months ago (before we knew about Tracker). Richard Su made lot of
work in this area (CCing him) so it would worh discuss this with him,
wiki page with related stuff is here:
https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Background_Processing
mail thread is here:
https://fedorahosted.org/pipermail/aeolus-devel/2012-April/009923.html
> David
>
>