You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Kishan <ki...@gmail.com> on 2021/05/13 05:39:19 UTC

Execute Ignite Callable Jobs with set priorities

Hi All,

I have a use case I need to create 1000s of ignite Callable tasks and
execute these tasks on ignite cluster. Some of these jobs should be executed
on high priority by ignite cluster. For example, ignite client has sent
around 50 tasks to cluster and now they are in queue or are being executed.
At this point, client receives a request which should be executed with
highest priority. Client will create a compute task with priority set to
High and send it to ignite cluster via executor service. Is there any way
ignite can know that certain tasks which are are submitted with high
priority should be executed before any other tasks present in the queue?

Thanks..
K



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Execute Ignite Callable Jobs with set priorities

Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!

I don't think so.

Regards,
-- 
Ilya Kasnacheev


вт, 25 мая 2021 г. в 14:36, Krish <ki...@gmail.com>:

> ilya.kasnacheev wrote
> > This means that Ignite prioritizing is a poor fit for you and you may
> need
> > to roll out your own, perhaps based on IgniteQueue.
>
> Does ignite provide priority queue implementation of IgniteQueue interface?
>
> Thanks,
> Krish
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Execute Ignite Callable Jobs with set priorities

Posted by Krish <ki...@gmail.com>.
ilya.kasnacheev wrote
> This means that Ignite prioritizing is a poor fit for you and you may need
> to roll out your own, perhaps based on IgniteQueue.

Does ignite provide priority queue implementation of IgniteQueue interface?

Thanks,
Krish



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Execute Ignite Callable Jobs with set priorities

Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!

It just sounds that you want to prioritize some other resource rather than
Ignite compute capacity, if you want cluster-wide priorities.

This means that Ignite prioritizing is a poor fit for you and you may need
to roll out your own, perhaps based on IgniteQueue.

Regards,
-- 
Ilya Kasnacheev


вс, 16 мая 2021 г. в 13:20, Krish <ki...@gmail.com>:

> Thanks Stephen and Ilya,
>
> As mentioned in job scheduling documentation, collisionAPI will take care
> of
> job scheduling when jobs arrive at destination node. Lets say I use
> PriorityQueueCollisionSpi and send three jobs with priorities 5, 7 and 10
> to
> one node, then that node will execute job with priority 10 first then job
> with priority 7 and then 5.
>
> However, My use case is different than this. Going with above example, in
> my
> use case, ignite client will send three jobs to three different nodes and I
> would still want these jobs to be executed based on their priorities.
> Basically, no matter how these jobs are distributed across the cluster they
> should be executed based on priority. Can this be achieved using
> CollisionAPI?
>
> Many Thanks,
> K
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>

Re: Execute Ignite Callable Jobs with set priorities

Posted by Stephen Darlington <st...@gridgain.com>.
> As mentioned in job scheduling documentation, collisionAPI will take care of
> job scheduling when jobs arrive at destination node. Lets say I use
> PriorityQueueCollisionSpi and send three jobs with priorities 5, 7 and 10 to
> one node, then that node will execute job with priority 10 first then job
> with priority 7 and then 5.

In fact, no. Assuming there are no other compute jobs, it will probably run all three jobs together. You can think of CollisionSPI answering the question “if I have a spare thread to execute some code, which job should I execute next?"

If the jobs are dependent on one another, you can use compute tasks (https://ignite.apache.org/docs/latest/distributed-computing/map-reduce <https://ignite.apache.org/docs/latest/distributed-computing/map-reduce>).

> Basically, no matter how these jobs are distributed across the cluster they
> should be executed based on priority. Can this be achieved using
> CollisionAPI?

That would be very inefficient. You’d potentially have lots of wasted capacity and there’s a high risk of starvation.

So there’s no out-of-the-box way to do that. You could manage it from the client side or write your own load balancing / collision algorithms. They’re both SPIs so it can be done.

Regards,
Stephen

Re: Execute Ignite Callable Jobs with set priorities

Posted by Krish <ki...@gmail.com>.
Thanks Stephen and Ilya,

As mentioned in job scheduling documentation, collisionAPI will take care of
job scheduling when jobs arrive at destination node. Lets say I use
PriorityQueueCollisionSpi and send three jobs with priorities 5, 7 and 10 to
one node, then that node will execute job with priority 10 first then job
with priority 7 and then 5.

However, My use case is different than this. Going with above example, in my
use case, ignite client will send three jobs to three different nodes and I
would still want these jobs to be executed based on their priorities.
Basically, no matter how these jobs are distributed across the cluster they
should be executed based on priority. Can this be achieved using
CollisionAPI?

Many Thanks,
K



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Execute Ignite Callable Jobs with set priorities

Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!

Just make sure to use
new PriorityQueueCollisionSpi().setStarvationPreventionEnabled(false);

otherwise you may get sorting errors on newer JVMs.

Regards,
-- 
Ilya Kasnacheev



чт, 13 мая 2021 г. в 11:31, Stephen Darlington <
stephen.darlington@gridgain.com>:

> Yes, it’s configurable using the CollisionSPI. More here:
> https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling
>
> Regards,
> Stephen
>
> > On 13 May 2021, at 06:39, Kishan <ki...@gmail.com> wrote:
> >
> > Hi All,
> >
> > I have a use case I need to create 1000s of ignite Callable tasks and
> > execute these tasks on ignite cluster. Some of these jobs should be
> executed
> > on high priority by ignite cluster. For example, ignite client has sent
> > around 50 tasks to cluster and now they are in queue or are being
> executed.
> > At this point, client receives a request which should be executed with
> > highest priority. Client will create a compute task with priority set to
> > High and send it to ignite cluster via executor service. Is there any way
> > ignite can know that certain tasks which are are submitted with high
> > priority should be executed before any other tasks present in the queue?
> >
> > Thanks..
> > K
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>
>

Re: Execute Ignite Callable Jobs with set priorities

Posted by Stephen Darlington <st...@gridgain.com>.
Yes, it’s configurable using the CollisionSPI. More here: https://ignite.apache.org/docs/latest/distributed-computing/job-scheduling

Regards,
Stephen

> On 13 May 2021, at 06:39, Kishan <ki...@gmail.com> wrote:
> 
> Hi All,
> 
> I have a use case I need to create 1000s of ignite Callable tasks and
> execute these tasks on ignite cluster. Some of these jobs should be executed
> on high priority by ignite cluster. For example, ignite client has sent
> around 50 tasks to cluster and now they are in queue or are being executed.
> At this point, client receives a request which should be executed with
> highest priority. Client will create a compute task with priority set to
> High and send it to ignite cluster via executor service. Is there any way
> ignite can know that certain tasks which are are submitted with high
> priority should be executed before any other tasks present in the queue?
> 
> Thanks..
> K
> 
> 
> 
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/