You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by djm132 <do...@gmail.com> on 2016/05/06 08:23:11 UTC
Async workers
Hi,
I have some confusion with implementing async task execution on cluster.
There is own API server embedded along with ignite core in one application.
API users execute http calls to some host as entry point and can be switched
to another one on failover.
1) REST API server accepts client requests for some long-running job.
2) After receiving API request there is a call to compute.run() and then I
am getting task session id from compute.future().
3) Task session id returned to user as API response.
4) As I understand, this run() call directed by load balancer to some node.
5) After some time, user requests API with previous taskId about task
status and/or result.
The problem here - how can I get task state/result using only taskId from
ANY cluster node (not initiated one) ?
Thanks.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Async-workers-tp4813.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Async workers
Posted by djm132 <do...@gmail.com>.
Thanks for pointing me to ComputeJobMasterLeaveAware interface.
I've implemented this the following way:
1) Created TaskInfo object with task parameters object, status, result, task
class name, etc.
2) During node selection in map method, taskInfo saved to cache with master
node id, assigned node id and status "running" (CPU adaptive load balancer
used).
3) Execute task on some node, get results, wrap exceptions to taskInfo's
task result field.
4) Initiator node receives callback from task future object and updates
cache with taskInfo changed to finished or failed state.
5) If master node dies, no callbacks issued of course and taskInfo remains
in running state.
6) After receiving client API request for task status/result I am using this
code to ping initiator node (only running state checked)
if (!Engine.isNodeAlive(taskInfo.masterNodeId)) {
Class<? extends Task<Object,TaskResult>> taskClass =
(Class<? extends Task<Object,TaskResult>>)Class.forName(taskInfo.taskClass);
taskInfo = Engine.addTask(taskClass, taskInfo.params);
}
7) Updated task info goes to cache with running status.
8) All finished or failed taskInfo's removed from cache on first client API
request or after 1 hour of no access (probably will leave only time
expiration).
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Async-workers-tp4813p4914.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Async workers
Posted by Alexei Scherbakov <al...@gmail.com>.
Hi,
If the master leaves, tasks will still continue to execute.
If your tasks need to be aware of the master state you should implement
ComputeJobMasterLeaveAware interface.
Considering the case I see two ways:
1) Put the task state into a cache where the key is task id and the value
is task state.
Periodically scan the cache for completed tasks and do cleanup.
This is similar to your solution.
Service API is not required but can be used.
2) Subscribe to TaskEvent s
The flaw of this solution is what event not guaranteed to be delivered if
the listener dies.
2016-05-06 15:16 GMT+03:00 djm132 <do...@gmail.com>:
> Thanks for clarification.
>
> So if the originator dies - all started tasks dies too, right?
>
> In this case I see solution like this:
>
> 1) Create task on initiator node.
> 2) Put node id + task data to ignite cache.
> 3) Create service which returns task liveness/status/result for taskId
> originated locally.
> 4) Create recurring job on every node which looks in cache and for every
> task found checks liveness of task. Then in case of error just fire task
> again.
> 5) When API asks for task state it looks to cache for taskId, finds node
> and
> ask it via service. Then cache cleared if task completed.
>
> May be there is better way to do it ?
>
> Thanks.
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Async-workers-tp4813p4829.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Alexei Scherbakov
Re: Async workers
Posted by djm132 <do...@gmail.com>.
Thanks for clarification.
So if the originator dies - all started tasks dies too, right?
In this case I see solution like this:
1) Create task on initiator node.
2) Put node id + task data to ignite cache.
3) Create service which returns task liveness/status/result for taskId
originated locally.
4) Create recurring job on every node which looks in cache and for every
task found checks liveness of task. Then in case of error just fire task
again.
5) When API asks for task state it looks to cache for taskId, finds node and
ask it via service. Then cache cleared if task completed.
May be there is better way to do it ?
Thanks.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Async-workers-tp4813p4829.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Async workers
Posted by Denis Magda <dm...@gridgain.com>.
Hi,
There is no way to get a task state from every node. The state of a particular task can be taken only on the node that initiated its execution and only if the task is still running. When a task is completed you have to store its result somewhere and pass to the client later.
In general if the task is still running you should execute IgniteCompute.activeTaskFutures on the originating node and check if it’s still in progress or not.
Makes sense?
—
Denis
> On May 6, 2016, at 11:23 AM, djm132 <do...@gmail.com> wrote:
>
> Hi,
>
> I have some confusion with implementing async task execution on cluster.
> There is own API server embedded along with ignite core in one application.
> API users execute http calls to some host as entry point and can be switched
> to another one on failover.
>
> 1) REST API server accepts client requests for some long-running job.
> 2) After receiving API request there is a call to compute.run() and then I
> am getting task session id from compute.future().
> 3) Task session id returned to user as API response.
> 4) As I understand, this run() call directed by load balancer to some node.
> 5) After some time, user requests API with previous taskId about task
> status and/or result.
>
> The problem here - how can I get task state/result using only taskId from
> ANY cluster node (not initiated one) ?
>
> Thanks.
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Async-workers-tp4813.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.