JobQueue provider ?

[Ian Boston <ie...@tfd.co.uk>]

Hi,
I see that the JobQueues are hard coded into the JobManager.

Would it be possible to make the JobQueue an API so that its could be
provided by another bundle ?

Best Regards
Ian

Re: JobQueue provider ?

[Ian Boston <ie...@tfd.co.uk>]

On 4 September 2013 10:30, Carsten Ziegeler <cz...@apache.org> wrote:
> Yes, I see your point - and partially agree :) However, the notion of a
> queue is slightly misleading here - in fact, the real queue is the content
> stored at a specific location in the resource tree - the location includes
> the sling instance id, the job topic.
> The java queue implementation is "just" the execution queue on a single
> instance - so for example if we would provide an implementation executing
> jobs via JMS, this wouldn't be a new queue implementation, but would
> require a different distribution mechanism.
> In fact, I plan to get away with the queue implementations completely and
> just have the queues in the resource tree with a simple implementation just
> picking the "next" job resource from the resource tree and executing it.

If the content protocol for picking a job and executing it could be a
public "api" then anyone wanting to do something odd could find the
next job of interest, pick it and execute it. When you get to
implementing the standard mechanism, if you could bear that in mind it
would be good. Hopefully considering that wont make things more
complicated than they need to be.

Ian


>
> Carsten
>
>
> 2013/9/4 Ian Boston <ie...@tfd.co.uk>
>
>> Hi,
>> Thanks for taking the time to explain the approach/work arround, I'll
>> give it a go.
>> I still think that it would be better to provide some mechanism to add
>> a queue behaviour without requiring the bundle to be re-released, as I
>> don't think we can predict every downstream use case in advance.
>> However I don't have a concrete requirement at this stage.
>> Best Regards
>> Ian
>>
>> On 4 September 2013 08:53, Carsten Ziegeler <cz...@apache.org> wrote:
>> > Yeah, maybe - in fact it's a whitelist and a blacklist or
>> > inclusion/exclusion list :)
>> > In any case, we released the module already - so the name will stick :)
>> >
>> > Carsten
>> >
>> >
>> > 2013/9/4 Bertrand Delacretaz <bd...@apache.org>
>> >
>> >> On Tue, Sep 3, 2013 at 7:43 PM, Carsten Ziegeler <cz...@apache.org>
>> >> wrote:
>> >> > ...actually black listing is exactly for these use cases. It gives you
>> >> > control which instance is able to handle what jobs. So you can
>> exclude a
>> >> > specific instance from processing a specific job, in your use case you
>> >> > exclude all but one.
>> >> > This is not blacklisting of event handlers if you're refering to that
>> >> one...
>> >>
>> >> So maybe blacklisting is not the right term in the context of jobs?
>> >>
>> >> IIUC this setting just causes some jobs to be ignored on some nodes,
>> >> so it could be just "ignored" instead of "blacklisted".
>> >>
>> >> -Bertrand
>> >>
>> >
>> >
>> >
>> > --
>> > Carsten Ziegeler
>> > cziegeler@apache.org
>>
>
>
>
> --
> Carsten Ziegeler
> cziegeler@apache.org

Re: JobQueue provider ?

[Carsten Ziegeler <cz...@apache.org>]

Yes, I see your point - and partially agree :) However, the notion of a
queue is slightly misleading here - in fact, the real queue is the content
stored at a specific location in the resource tree - the location includes
the sling instance id, the job topic.
The java queue implementation is "just" the execution queue on a single
instance - so for example if we would provide an implementation executing
jobs via JMS, this wouldn't be a new queue implementation, but would
require a different distribution mechanism.
In fact, I plan to get away with the queue implementations completely and
just have the queues in the resource tree with a simple implementation just
picking the "next" job resource from the resource tree and executing it.

Carsten


2013/9/4 Ian Boston <ie...@tfd.co.uk>

> Hi,
> Thanks for taking the time to explain the approach/work arround, I'll
> give it a go.
> I still think that it would be better to provide some mechanism to add
> a queue behaviour without requiring the bundle to be re-released, as I
> don't think we can predict every downstream use case in advance.
> However I don't have a concrete requirement at this stage.
> Best Regards
> Ian
>
> On 4 September 2013 08:53, Carsten Ziegeler <cz...@apache.org> wrote:
> > Yeah, maybe - in fact it's a whitelist and a blacklist or
> > inclusion/exclusion list :)
> > In any case, we released the module already - so the name will stick :)
> >
> > Carsten
> >
> >
> > 2013/9/4 Bertrand Delacretaz <bd...@apache.org>
> >
> >> On Tue, Sep 3, 2013 at 7:43 PM, Carsten Ziegeler <cz...@apache.org>
> >> wrote:
> >> > ...actually black listing is exactly for these use cases. It gives you
> >> > control which instance is able to handle what jobs. So you can
> exclude a
> >> > specific instance from processing a specific job, in your use case you
> >> > exclude all but one.
> >> > This is not blacklisting of event handlers if you're refering to that
> >> one...
> >>
> >> So maybe blacklisting is not the right term in the context of jobs?
> >>
> >> IIUC this setting just causes some jobs to be ignored on some nodes,
> >> so it could be just "ignored" instead of "blacklisted".
> >>
> >> -Bertrand
> >>
> >
> >
> >
> > --
> > Carsten Ziegeler
> > cziegeler@apache.org
>



-- 
Carsten Ziegeler
cziegeler@apache.org

Re: JobQueue provider ?

[Ian Boston <ie...@tfd.co.uk>]

Hi,
Thanks for taking the time to explain the approach/work arround, I'll
give it a go.
I still think that it would be better to provide some mechanism to add
a queue behaviour without requiring the bundle to be re-released, as I
don't think we can predict every downstream use case in advance.
However I don't have a concrete requirement at this stage.
Best Regards
Ian

On 4 September 2013 08:53, Carsten Ziegeler <cz...@apache.org> wrote:
> Yeah, maybe - in fact it's a whitelist and a blacklist or
> inclusion/exclusion list :)
> In any case, we released the module already - so the name will stick :)
>
> Carsten
>
>
> 2013/9/4 Bertrand Delacretaz <bd...@apache.org>
>
>> On Tue, Sep 3, 2013 at 7:43 PM, Carsten Ziegeler <cz...@apache.org>
>> wrote:
>> > ...actually black listing is exactly for these use cases. It gives you
>> > control which instance is able to handle what jobs. So you can exclude a
>> > specific instance from processing a specific job, in your use case you
>> > exclude all but one.
>> > This is not blacklisting of event handlers if you're refering to that
>> one...
>>
>> So maybe blacklisting is not the right term in the context of jobs?
>>
>> IIUC this setting just causes some jobs to be ignored on some nodes,
>> so it could be just "ignored" instead of "blacklisted".
>>
>> -Bertrand
>>
>
>
>
> --
> Carsten Ziegeler
> cziegeler@apache.org

Re: JobQueue provider ?

[Carsten Ziegeler <cz...@apache.org>]

Yeah, maybe - in fact it's a whitelist and a blacklist or
inclusion/exclusion list :)
In any case, we released the module already - so the name will stick :)

Carsten


2013/9/4 Bertrand Delacretaz <bd...@apache.org>

> On Tue, Sep 3, 2013 at 7:43 PM, Carsten Ziegeler <cz...@apache.org>
> wrote:
> > ...actually black listing is exactly for these use cases. It gives you
> > control which instance is able to handle what jobs. So you can exclude a
> > specific instance from processing a specific job, in your use case you
> > exclude all but one.
> > This is not blacklisting of event handlers if you're refering to that
> one...
>
> So maybe blacklisting is not the right term in the context of jobs?
>
> IIUC this setting just causes some jobs to be ignored on some nodes,
> so it could be just "ignored" instead of "blacklisted".
>
> -Bertrand
>



-- 
Carsten Ziegeler
cziegeler@apache.org

Re: JobQueue provider ?

[Bertrand Delacretaz <bd...@apache.org>]

On Tue, Sep 3, 2013 at 7:43 PM, Carsten Ziegeler <cz...@apache.org> wrote:
> ...actually black listing is exactly for these use cases. It gives you
> control which instance is able to handle what jobs. So you can exclude a
> specific instance from processing a specific job, in your use case you
> exclude all but one.
> This is not blacklisting of event handlers if you're refering to that one...

So maybe blacklisting is not the right term in the context of jobs?

IIUC this setting just causes some jobs to be ignored on some nodes,
so it could be just "ignored" instead of "blacklisted".

-Bertrand

Re: JobQueue provider ?

[Carsten Ziegeler <cz...@apache.org>]

Yes, I think this would really make the whole implementation and handling
much more compicated.
And actually black listing is exactly for these use cases. It gives you
control which instance is able to handle what jobs. So you can exclude a
specific instance from processing a specific job, in your use case you
exclude all but one.
This is not blacklisting of event handlers if you're refering to that one.

Carsten


2013/9/3 Ian Boston <ie...@tfd.co.uk>

> Hi,
> I see how it could work, but it seems an odd way of making a queue
> work, to actively blacklist a topic queue on certain instances. I
> thought black listing was intended for non responsive consumers.
>
> Is there something inherently wrong with the concept of a JobQueue
> provider ?
> Is it because it would make the implementation of the JobManager too
> complex where providers are not present at activation ?
>
> Best Regards
> Ian
>
> On 3 September 2013 19:00, Carsten Ziegeler <cz...@apache.org> wrote:
> > I think this is beyond queuing - you can use a topology listener for that
> > by disabling a topic on all instances but the active one. If the active
> one
> > goes down, an event is fired on the remaining instances, and another one
> > can take over. This will ensure rescheduling of jobs to that instance by
> > the job manager.
> > So all you have to take care of is managing the blacklist of the job
> > consumer manager based on topology events.
> >
> > Carsten
> >
> >
> > 2013/9/3 Ian Boston <ie...@tfd.co.uk>
> >
> >> Hi,
> >>
> >> I want to be able to submit jobs, distribute those jobs and have those
> >> jobs restart on instance failover in a way not supported by the
> >> current jobs queues in the event bundle, without requiring the event
> >> bundle to be re-released.
> >>
> >> eg
> >> Run only on the active Sling instance in a active/passive pair and if
> >> the active instance changes, then that instance new active instance to
> >> run all the jobs, including jobs that were added, but not run on the
> >> old active instance.
> >>
> >> Ian
> >>
> >>
> >> On 3 September 2013 18:00, Carsten Ziegeler <cz...@apache.org>
> wrote:
> >> > In theory this would be possible, but I guess this would create maybe
> >> more
> >> > problems - an implementation might not be available (anymore), a
> >> > configuration might point to a non existing implementation etc. A lot
> >> more
> >> > things could go wrong.
> >> >
> >> >
> >> > What's your use case? :)
> >> >
> >> > Carsten
> >> >
> >> >
> >> > 2013/9/3 Ian Boston <ie...@tfd.co.uk>
> >> >
> >> >> Hi,
> >> >> I see that the JobQueues are hard coded into the JobManager.
> >> >>
> >> >> Would it be possible to make the JobQueue an API so that its could be
> >> >> provided by another bundle ?
> >> >>
> >> >> Best Regards
> >> >> Ian
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Carsten Ziegeler
> >> > cziegeler@apache.org
> >>
> >
> >
> >
> > --
> > Carsten Ziegeler
> > cziegeler@apache.org
>



-- 
Carsten Ziegeler
cziegeler@apache.org

Re: JobQueue provider ?

[Ian Boston <ie...@tfd.co.uk>]

Hi,
I see how it could work, but it seems an odd way of making a queue
work, to actively blacklist a topic queue on certain instances. I
thought black listing was intended for non responsive consumers.

Is there something inherently wrong with the concept of a JobQueue provider ?
Is it because it would make the implementation of the JobManager too
complex where providers are not present at activation ?

Best Regards
Ian

On 3 September 2013 19:00, Carsten Ziegeler <cz...@apache.org> wrote:
> I think this is beyond queuing - you can use a topology listener for that
> by disabling a topic on all instances but the active one. If the active one
> goes down, an event is fired on the remaining instances, and another one
> can take over. This will ensure rescheduling of jobs to that instance by
> the job manager.
> So all you have to take care of is managing the blacklist of the job
> consumer manager based on topology events.
>
> Carsten
>
>
> 2013/9/3 Ian Boston <ie...@tfd.co.uk>
>
>> Hi,
>>
>> I want to be able to submit jobs, distribute those jobs and have those
>> jobs restart on instance failover in a way not supported by the
>> current jobs queues in the event bundle, without requiring the event
>> bundle to be re-released.
>>
>> eg
>> Run only on the active Sling instance in a active/passive pair and if
>> the active instance changes, then that instance new active instance to
>> run all the jobs, including jobs that were added, but not run on the
>> old active instance.
>>
>> Ian
>>
>>
>> On 3 September 2013 18:00, Carsten Ziegeler <cz...@apache.org> wrote:
>> > In theory this would be possible, but I guess this would create maybe
>> more
>> > problems - an implementation might not be available (anymore), a
>> > configuration might point to a non existing implementation etc. A lot
>> more
>> > things could go wrong.
>> >
>> >
>> > What's your use case? :)
>> >
>> > Carsten
>> >
>> >
>> > 2013/9/3 Ian Boston <ie...@tfd.co.uk>
>> >
>> >> Hi,
>> >> I see that the JobQueues are hard coded into the JobManager.
>> >>
>> >> Would it be possible to make the JobQueue an API so that its could be
>> >> provided by another bundle ?
>> >>
>> >> Best Regards
>> >> Ian
>> >>
>> >
>> >
>> >
>> > --
>> > Carsten Ziegeler
>> > cziegeler@apache.org
>>
>
>
>
> --
> Carsten Ziegeler
> cziegeler@apache.org

Re: JobQueue provider ?

[Carsten Ziegeler <cz...@apache.org>]

I think this is beyond queuing - you can use a topology listener for that
by disabling a topic on all instances but the active one. If the active one
goes down, an event is fired on the remaining instances, and another one
can take over. This will ensure rescheduling of jobs to that instance by
the job manager.
So all you have to take care of is managing the blacklist of the job
consumer manager based on topology events.

Carsten


2013/9/3 Ian Boston <ie...@tfd.co.uk>

> Hi,
>
> I want to be able to submit jobs, distribute those jobs and have those
> jobs restart on instance failover in a way not supported by the
> current jobs queues in the event bundle, without requiring the event
> bundle to be re-released.
>
> eg
> Run only on the active Sling instance in a active/passive pair and if
> the active instance changes, then that instance new active instance to
> run all the jobs, including jobs that were added, but not run on the
> old active instance.
>
> Ian
>
>
> On 3 September 2013 18:00, Carsten Ziegeler <cz...@apache.org> wrote:
> > In theory this would be possible, but I guess this would create maybe
> more
> > problems - an implementation might not be available (anymore), a
> > configuration might point to a non existing implementation etc. A lot
> more
> > things could go wrong.
> >
> >
> > What's your use case? :)
> >
> > Carsten
> >
> >
> > 2013/9/3 Ian Boston <ie...@tfd.co.uk>
> >
> >> Hi,
> >> I see that the JobQueues are hard coded into the JobManager.
> >>
> >> Would it be possible to make the JobQueue an API so that its could be
> >> provided by another bundle ?
> >>
> >> Best Regards
> >> Ian
> >>
> >
> >
> >
> > --
> > Carsten Ziegeler
> > cziegeler@apache.org
>



-- 
Carsten Ziegeler
cziegeler@apache.org

Re: JobQueue provider ?

[Ian Boston <ie...@tfd.co.uk>]

Hi,

I want to be able to submit jobs, distribute those jobs and have those
jobs restart on instance failover in a way not supported by the
current jobs queues in the event bundle, without requiring the event
bundle to be re-released.

eg
Run only on the active Sling instance in a active/passive pair and if
the active instance changes, then that instance new active instance to
run all the jobs, including jobs that were added, but not run on the
old active instance.

Ian


On 3 September 2013 18:00, Carsten Ziegeler <cz...@apache.org> wrote:
> In theory this would be possible, but I guess this would create maybe more
> problems - an implementation might not be available (anymore), a
> configuration might point to a non existing implementation etc. A lot more
> things could go wrong.
>
>
> What's your use case? :)
>
> Carsten
>
>
> 2013/9/3 Ian Boston <ie...@tfd.co.uk>
>
>> Hi,
>> I see that the JobQueues are hard coded into the JobManager.
>>
>> Would it be possible to make the JobQueue an API so that its could be
>> provided by another bundle ?
>>
>> Best Regards
>> Ian
>>
>
>
>
> --
> Carsten Ziegeler
> cziegeler@apache.org

Re: JobQueue provider ?

[Carsten Ziegeler <cz...@apache.org>]

In theory this would be possible, but I guess this would create maybe more
problems - an implementation might not be available (anymore), a
configuration might point to a non existing implementation etc. A lot more
things could go wrong.


What's your use case? :)

Carsten


2013/9/3 Ian Boston <ie...@tfd.co.uk>

> Hi,
> I see that the JobQueues are hard coded into the JobManager.
>
> Would it be possible to make the JobQueue an API so that its could be
> provided by another bundle ?
>
> Best Regards
> Ian
>



-- 
Carsten Ziegeler
cziegeler@apache.org