You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Alexei Scherbakov <al...@gmail.com> on 2017/01/26 20:36:37 UTC
Transactional tasks
Guys,
How difficult would be to support transactional tasks?
What means every job in task executed in it's own transaction.
In case of single job failure or reduce phase failure all transaction
started by jobs are rolled back.
Only if all jobs are successfully executed, corresponding transactions are
commited.
Also it would be very desirable to implement tasks failover in the similar
way how jobs failover is implemented.
In case of master's failure jobs are rolled back, and task is restarted on
another node.
This should greatly simplify implementing complex business processes.
--
Best regards,
Alexei Scherbakov
Re: Transactional tasks
Posted by Yakov Zhdanov <yz...@apache.org>.
>> This is prefectly valid. Second transaction'll wait for completion of
first.
Alex S., I think you did not get the point. It seems AG meant two jobs of
the same task, so it is one TX. It is unclear to me how to resolve this
automatically. I would agree with AG and permit only primary keys local to
the node job is running on.
>> Could you provide more details? Current job transactions must be rolled back
all together and task must be routed to other node.
AG, this sounds like a solution? Can we put hard requirement here - task TX
does not survive node failure before prepare completes?
--Yakov
2017-01-30 23:55 GMT+03:00 Alexei Scherbakov <al...@gmail.com>:
> Alexey,
>
> Every job is executed in transaction having same semantics as any other
> transaction, but transaction is managed by Ignite transparently.
>
> Job's transaction is rolled back on throwing any unchecked exception or
> special TransactionRollbackException.
>
> Concurrency issues between jobs are solved using entry locking or currency
> primitives.
>
> I think the closest similar thing is JTA transaction enlisting. Maybe I
> even'll implement such logic using current Ignite JTA module, haven't tried
> yet. But I don't like JTA overhead and a requirement to have JTA
> coordinator.
>
> 2017-01-27 13:40 GMT+03:00 Alexey Goncharuk <al...@gmail.com>:
>
> > Alexei,
> >
> > Can you describe the semantics of the task in greater detail?
> > * What if two jobs on different nodes access and try to put the same
> key?
> > (this can be resolved by allowing a job only access local primary keys)
> >
>
> This is prefectly valid. Second transaction'll wait for completion of
> first.
>
>
> > * How do you define the lock acquisition order and prevent deadlocks? I
> > assume that running such a compute task will involve a lot of keys, so it
> > is quite unlikely that OPTIMISTIC SERIALIZABLE transactions are
> applicable
> > here
> >
>
> Same as any other transaction - user is responsible for avoiding deadlock.
>
>
> > * Currently a started transaction locks the cache topology, so job
> > failover in the case of node crash will be very hard to implement
> >
>
> Could you provide more details? Current job transactions must be rolled
> back all together and task must be routed to other node.
>
>
> >
> > 2017-01-26 23:36 GMT+03:00 Alexei Scherbakov <
> alexey.scherbakoff@gmail.com
> > >:
> >
> > > Guys,
> > >
> > > How difficult would be to support transactional tasks?
> > >
> > > What means every job in task executed in it's own transaction.
> > >
> > > In case of single job failure or reduce phase failure all transaction
> > > started by jobs are rolled back.
> > >
> > > Only if all jobs are successfully executed, corresponding transactions
> > are
> > > commited.
> > >
> > > Also it would be very desirable to implement tasks failover in the
> > similar
> > > way how jobs failover is implemented.
> > >
> > > In case of master's failure jobs are rolled back, and task is restarted
> > on
> > > another node.
> > >
> > > This should greatly simplify implementing complex business processes.
> > >
> > > --
> > >
> > > Best regards,
> > > Alexei Scherbakov
> > >
> >
>
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>
Re: Transactional tasks
Posted by Alexei Scherbakov <al...@gmail.com>.
Alexey,
Every job is executed in transaction having same semantics as any other
transaction, but transaction is managed by Ignite transparently.
Job's transaction is rolled back on throwing any unchecked exception or
special TransactionRollbackException.
Concurrency issues between jobs are solved using entry locking or currency
primitives.
I think the closest similar thing is JTA transaction enlisting. Maybe I
even'll implement such logic using current Ignite JTA module, haven't tried
yet. But I don't like JTA overhead and a requirement to have JTA
coordinator.
2017-01-27 13:40 GMT+03:00 Alexey Goncharuk <al...@gmail.com>:
> Alexei,
>
> Can you describe the semantics of the task in greater detail?
> * What if two jobs on different nodes access and try to put the same key?
> (this can be resolved by allowing a job only access local primary keys)
>
This is prefectly valid. Second transaction'll wait for completion of first.
> * How do you define the lock acquisition order and prevent deadlocks? I
> assume that running such a compute task will involve a lot of keys, so it
> is quite unlikely that OPTIMISTIC SERIALIZABLE transactions are applicable
> here
>
Same as any other transaction - user is responsible for avoiding deadlock.
> * Currently a started transaction locks the cache topology, so job
> failover in the case of node crash will be very hard to implement
>
Could you provide more details? Current job transactions must be rolled
back all together and task must be routed to other node.
>
> 2017-01-26 23:36 GMT+03:00 Alexei Scherbakov <alexey.scherbakoff@gmail.com
> >:
>
> > Guys,
> >
> > How difficult would be to support transactional tasks?
> >
> > What means every job in task executed in it's own transaction.
> >
> > In case of single job failure or reduce phase failure all transaction
> > started by jobs are rolled back.
> >
> > Only if all jobs are successfully executed, corresponding transactions
> are
> > commited.
> >
> > Also it would be very desirable to implement tasks failover in the
> similar
> > way how jobs failover is implemented.
> >
> > In case of master's failure jobs are rolled back, and task is restarted
> on
> > another node.
> >
> > This should greatly simplify implementing complex business processes.
> >
> > --
> >
> > Best regards,
> > Alexei Scherbakov
> >
>
--
Best regards,
Alexei Scherbakov
Re: Transactional tasks
Posted by Alexey Goncharuk <al...@gmail.com>.
Alexei,
Can you describe the semantics of the task in greater detail?
* What if two jobs on different nodes access and try to put the same key?
(this can be resolved by allowing a job only access local primary keys)
* How do you define the lock acquisition order and prevent deadlocks? I
assume that running such a compute task will involve a lot of keys, so it
is quite unlikely that OPTIMISTIC SERIALIZABLE transactions are applicable
here
* Currently a started transaction locks the cache topology, so job
failover in the case of node crash will be very hard to implement
2017-01-26 23:36 GMT+03:00 Alexei Scherbakov <al...@gmail.com>:
> Guys,
>
> How difficult would be to support transactional tasks?
>
> What means every job in task executed in it's own transaction.
>
> In case of single job failure or reduce phase failure all transaction
> started by jobs are rolled back.
>
> Only if all jobs are successfully executed, corresponding transactions are
> commited.
>
> Also it would be very desirable to implement tasks failover in the similar
> way how jobs failover is implemented.
>
> In case of master's failure jobs are rolled back, and task is restarted on
> another node.
>
> This should greatly simplify implementing complex business processes.
>
> --
>
> Best regards,
> Alexei Scherbakov
>