You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sling.apache.org by Julian Sedding <js...@gmail.com> on 2015/09/24 09:53:50 UTC
Job Distribution
Hi all
I was looking into the following deployment scenario recently:
- Oak Mongo with 4 attached, clustered Sling nodes
- 2 nodes to be used to render the HTML UI
- 2 nodes to be used for background processing
The motivation was to remove background load from the rendering nodes
in order to improve user experience (responsiveness etc).
I got the load distribution working, however, there is one aspect I
think could be improved. Ordered Job queues get processed on the
leader instance.
I would like ordered Jobs to be processed on one of the processing
nodes. However, this cannot currently be achieved reliably, because I
cannot control which node is the leader. Especially in the case of a
failover situation.
Instead of mandating a leader, I thought it may be better to let the
leader nominate an "ordered-queue processing node". The leader could
then assign this role to a node that signals its availability for that
role. As a bonus, with such a setup, it may even be possible to
distribute the processing of different ordered queues to differenet
nodes.
So in my scenario, I would configure the "processing" nodes to be
available for processing ordered queues. I don't care any more which
node is the leader, because the leader would assign jobs from ordered
queues to one of the processing nodes.
WDYT? Is this a realistic scenario, or have I got any misconceptions
that would prevent such a setup?
Regards
Julian
Re: Job Distribution
Posted by Carsten Ziegeler <cz...@apache.org>.
Am 24.09.15 um 10:45 schrieb Stefan Egli:
> Hi,
>
> There are various options here: you could distribute the 'queue chiefs'
> via some hashing amongst all instances in the cluster view, and reassign
> those chiefs upon a topology change. Or you could let the leader
> explicitly send a 'become-chief' job/command to particular instances that
> would then blush and proudly start as chief of a particular queue - and a
> 'step-down-as-chief' command in case the leader detects it needs to
> redistribute load.
>
Right, I guess a hash based selection is easier as it avoids any
additional communication which then might require acks and so on.
Regards
Carsten
--
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org
Re: Job Distribution
Posted by Stefan Egli <st...@apache.org>.
Hi,
There are various options here: you could distribute the 'queue chiefs'
via some hashing amongst all instances in the cluster view, and reassign
those chiefs upon a topology change. Or you could let the leader
explicitly send a 'become-chief' job/command to particular instances that
would then blush and proudly start as chief of a particular queue - and a
'step-down-as-chief' command in case the leader detects it needs to
redistribute load.
Cheers,
Stefan
On 24/09/15 10:01, "Carsten Ziegeler" <cz...@apache.org> wrote:
>Hi Julian,
>
>I think that makes sense, it just adds one indirection on determining
>who will be processing ordered queues - but that shouldn't pose a problem.
>
>Regards
>Carsten
>
>Am 24.09.15 um 09:53 schrieb Julian Sedding:
>> Hi all
>>
>> I was looking into the following deployment scenario recently:
>>
>> - Oak Mongo with 4 attached, clustered Sling nodes
>> - 2 nodes to be used to render the HTML UI
>> - 2 nodes to be used for background processing
>>
>> The motivation was to remove background load from the rendering nodes
>> in order to improve user experience (responsiveness etc).
>>
>> I got the load distribution working, however, there is one aspect I
>> think could be improved. Ordered Job queues get processed on the
>> leader instance.
>>
>> I would like ordered Jobs to be processed on one of the processing
>> nodes. However, this cannot currently be achieved reliably, because I
>> cannot control which node is the leader. Especially in the case of a
>> failover situation.
>>
>> Instead of mandating a leader, I thought it may be better to let the
>> leader nominate an "ordered-queue processing node". The leader could
>> then assign this role to a node that signals its availability for that
>> role. As a bonus, with such a setup, it may even be possible to
>> distribute the processing of different ordered queues to differenet
>> nodes.
>>
>> So in my scenario, I would configure the "processing" nodes to be
>> available for processing ordered queues. I don't care any more which
>> node is the leader, because the leader would assign jobs from ordered
>> queues to one of the processing nodes.
>>
>> WDYT? Is this a realistic scenario, or have I got any misconceptions
>> that would prevent such a setup?
>>
>> Regards
>> Julian
>>
>
>
>--
>Carsten Ziegeler
>Adobe Research Switzerland
>cziegeler@apache.org
Re: Job Distribution
Posted by Carsten Ziegeler <cz...@apache.org>.
Hi Julian,
I think that makes sense, it just adds one indirection on determining
who will be processing ordered queues - but that shouldn't pose a problem.
Regards
Carsten
Am 24.09.15 um 09:53 schrieb Julian Sedding:
> Hi all
>
> I was looking into the following deployment scenario recently:
>
> - Oak Mongo with 4 attached, clustered Sling nodes
> - 2 nodes to be used to render the HTML UI
> - 2 nodes to be used for background processing
>
> The motivation was to remove background load from the rendering nodes
> in order to improve user experience (responsiveness etc).
>
> I got the load distribution working, however, there is one aspect I
> think could be improved. Ordered Job queues get processed on the
> leader instance.
>
> I would like ordered Jobs to be processed on one of the processing
> nodes. However, this cannot currently be achieved reliably, because I
> cannot control which node is the leader. Especially in the case of a
> failover situation.
>
> Instead of mandating a leader, I thought it may be better to let the
> leader nominate an "ordered-queue processing node". The leader could
> then assign this role to a node that signals its availability for that
> role. As a bonus, with such a setup, it may even be possible to
> distribute the processing of different ordered queues to differenet
> nodes.
>
> So in my scenario, I would configure the "processing" nodes to be
> available for processing ordered queues. I don't care any more which
> node is the leader, because the leader would assign jobs from ordered
> queues to one of the processing nodes.
>
> WDYT? Is this a realistic scenario, or have I got any misconceptions
> that would prevent such a setup?
>
> Regards
> Julian
>
--
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org