You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Dhruv Mahajan <dh...@gmail.com> on 2016/04/08 19:06:36 UTC
ICompletedTask semantics
Hi
I have a question regarding the semantics of ICompletedTask. Does this mean
that Dispose on the Task has already been called or it will be called after
driver acks. this message? I feel it is former but just wanted to verify.
Dhruv
Re: ICompletedTask semantics
Posted by Andrew Chung <af...@gmail.com>.
Or perhaps we can simply add a DisposeSucceeded field to
`ICompletedTask`? Hopefully no one has implemented it on their own.
Thanks,
Andrew
On Fri, Apr 8, 2016 at 2:04 PM, Julia Wang (QIUHE)
<Qi...@microsoft.com> wrote:
> We discussed the same topic in last Nov :)
>
> -----Original Message-----
> From: Tony Majestro [mailto:tmajest@microsoft.com]
> Sent: Friday, April 8, 2016 1:10 PM
> To: dev@reef.apache.org
> Subject: RE: ICompletedTask semantics
>
>> `ITask` has only one method, and as part of its exit, it needs to release all resources. Failure to do so would be a task failure.
>
> Even if ITask documents the above, it would be more idiomatic to just implement the IDisposable interface. It's a very explicit way of stating that this class might have resources that need cleaned up.
>
> -----Original Message-----
> From: Markus Weimer [mailto:markus@weimo.de]
> Sent: Friday, April 8, 2016 12:50 PM
> To: dev@reef.apache.org
> Subject: Re: ICompletedTask semantics
>
> On 2016-04-08 11:40, Andrew Chung wrote:
>> Examples include Dispose methods that modify states in external
>> systems, e.g. releasing distributed lock.
>
> Fair. However, I don't understand why `ITask` needs to be `Disposable` in the first place. `ITask` has only one method, and as part of its exit, it needs to release all resources. Failure to do so would be a task failure.
>
> In other words, I don't see anyone writing something like `using(var t = new MyTask()){...}`.
>
> Markus
RE: ICompletedTask semantics
Posted by "Julia Wang (QIUHE)" <Qi...@microsoft.com>.
We discussed the same topic in last Nov :)
-----Original Message-----
From: Tony Majestro [mailto:tmajest@microsoft.com]
Sent: Friday, April 8, 2016 1:10 PM
To: dev@reef.apache.org
Subject: RE: ICompletedTask semantics
> `ITask` has only one method, and as part of its exit, it needs to release all resources. Failure to do so would be a task failure.
Even if ITask documents the above, it would be more idiomatic to just implement the IDisposable interface. It's a very explicit way of stating that this class might have resources that need cleaned up.
-----Original Message-----
From: Markus Weimer [mailto:markus@weimo.de]
Sent: Friday, April 8, 2016 12:50 PM
To: dev@reef.apache.org
Subject: Re: ICompletedTask semantics
On 2016-04-08 11:40, Andrew Chung wrote:
> Examples include Dispose methods that modify states in external
> systems, e.g. releasing distributed lock.
Fair. However, I don't understand why `ITask` needs to be `Disposable` in the first place. `ITask` has only one method, and as part of its exit, it needs to release all resources. Failure to do so would be a task failure.
In other words, I don't see anyone writing something like `using(var t = new MyTask()){...}`.
Markus
RE: ICompletedTask semantics
Posted by Tony Majestro <tm...@microsoft.com>.
> `ITask` has only one method, and as part of its exit, it needs to release all resources. Failure to do so would be a task failure.
Even if ITask documents the above, it would be more idiomatic to just implement the IDisposable interface. It's a very explicit way of stating that this class might have resources that need cleaned up.
-----Original Message-----
From: Markus Weimer [mailto:markus@weimo.de]
Sent: Friday, April 8, 2016 12:50 PM
To: dev@reef.apache.org
Subject: Re: ICompletedTask semantics
On 2016-04-08 11:40, Andrew Chung wrote:
> Examples include Dispose methods that modify states in external
> systems, e.g. releasing distributed lock.
Fair. However, I don't understand why `ITask` needs to be `Disposable` in the first place. `ITask` has only one method, and as part of its exit, it needs to release all resources. Failure to do so would be a task failure.
In other words, I don't see anyone writing something like `using(var t = new MyTask()){...}`.
Markus
Re: ICompletedTask semantics
Posted by Markus Weimer <ma...@weimo.de>.
On 2016-04-08 11:40, Andrew Chung wrote:
> Examples include Dispose methods that modify states in external
> systems, e.g. releasing distributed lock.
Fair. However, I don't understand why `ITask` needs to be `Disposable`
in the first place. `ITask` has only one method, and as part of its
exit, it needs to release all resources. Failure to do so would be a
task failure.
In other words, I don't see anyone writing something like `using(var t =
new MyTask()){...}`.
Markus
Re: ICompletedTask semantics
Posted by Andrew Chung <af...@gmail.com>.
Examples include Dispose methods that modify states in external
systems, e.g. releasing distributed lock.
Thanks,
Andrew
On Fri, Apr 8, 2016 at 11:29 AM, Dhruv Mahajan <dh...@gmail.com> wrote:
> I feel if dispose fails, Task should still be termed as completed and
> failure should be ignored. Can we think of scenarios in which this logic is
> bad?
>
> Dhruv
>
> On Fri, Apr 8, 2016 at 11:18 AM, Andrew Chung <af...@gmail.com> wrote:
>
>> Hi Dhruv,
>>
>> Interesting question. There might actually be a race condition here
>> (somewhat, since I'm not sure if we count the `Dispose` call as part
>> of the Task), given that `Dispose` is actually called *after* setting
>> the Task status to `Completed`, and the Task thread locks on a
>> separate object than the Heartbeat thread. Only the Task status set
>> operation is atomic. Thus, if the Heartbeat thread gets the Task
>> status and sends the message to the Driver prior to the Dispose is
>> called, it is possible that Task.Dispose is not yet called before the
>> Driver receives the Task completion message.
>>
>> Given that we haven't defined a particular order for this sequence of
>> events, perhaps we should change the behavior to what Dhruv expects?
>>
>> Another thing to consider is whether we count `Dispose` as part of the
>> Task. e.g. If Dispose fails, do we count the Task as failed (Somewhat
>> dangerous, since many users don't implement Dispose on their Tasks)?
>> Do we only count the Task as Completed after the Dispose is done? What
>> do you think?
>>
>> Thanks,
>> Andrew
>>
>> On Fri, Apr 8, 2016 at 10:26 AM, Julia Wang (QIUHE)
>> <Qi...@microsoft.com> wrote:
>> > It has been disposed I believe.
>> >
>> > -----Original Message-----
>> > From: Dhruv Mahajan [mailto:dhruv.mahajan@gmail.com]
>> > Sent: Friday, April 8, 2016 10:07 AM
>> > To: dev@reef.apache.org
>> > Subject: ICompletedTask semantics
>> >
>> > Hi
>> >
>> > I have a question regarding the semantics of ICompletedTask. Does this
>> mean that Dispose on the Task has already been called or it will be called
>> after driver acks. this message? I feel it is former but just wanted to
>> verify.
>> >
>> > Dhruv
>>
RE: ICompletedTask semantics
Posted by "Julia Wang (QIUHE)" <Qi...@microsoft.com>.
I agree. If task is completed, we should try to dispose task. But if dispose fails, we should ignore it.
After driver receives task completed event, driver doesn't acknowledge evaluator, right? It is evaluator, that finally dispose the task. The sequence 1. Set task status to Complete 2. dispose task (ignore error if any) 3. Then send event back to driver seems more reasonable to me if the change is not hard.
Julia
-----Original Message-----
From: Dhruv Mahajan [mailto:dhruv.mahajan@gmail.com]
Sent: Friday, April 8, 2016 11:29 AM
To: dev@reef.apache.org
Subject: Re: ICompletedTask semantics
I feel if dispose fails, Task should still be termed as completed and failure should be ignored. Can we think of scenarios in which this logic is bad?
Dhruv
On Fri, Apr 8, 2016 at 11:18 AM, Andrew Chung <af...@gmail.com> wrote:
> Hi Dhruv,
>
> Interesting question. There might actually be a race condition here
> (somewhat, since I'm not sure if we count the `Dispose` call as part
> of the Task), given that `Dispose` is actually called *after* setting
> the Task status to `Completed`, and the Task thread locks on a
> separate object than the Heartbeat thread. Only the Task status set
> operation is atomic. Thus, if the Heartbeat thread gets the Task
> status and sends the message to the Driver prior to the Dispose is
> called, it is possible that Task.Dispose is not yet called before the
> Driver receives the Task completion message.
>
> Given that we haven't defined a particular order for this sequence of
> events, perhaps we should change the behavior to what Dhruv expects?
>
> Another thing to consider is whether we count `Dispose` as part of the
> Task. e.g. If Dispose fails, do we count the Task as failed (Somewhat
> dangerous, since many users don't implement Dispose on their Tasks)?
> Do we only count the Task as Completed after the Dispose is done? What
> do you think?
>
> Thanks,
> Andrew
>
> On Fri, Apr 8, 2016 at 10:26 AM, Julia Wang (QIUHE)
> <Qi...@microsoft.com> wrote:
> > It has been disposed I believe.
> >
> > -----Original Message-----
> > From: Dhruv Mahajan [mailto:dhruv.mahajan@gmail.com]
> > Sent: Friday, April 8, 2016 10:07 AM
> > To: dev@reef.apache.org
> > Subject: ICompletedTask semantics
> >
> > Hi
> >
> > I have a question regarding the semantics of ICompletedTask. Does
> > this
> mean that Dispose on the Task has already been called or it will be
> called after driver acks. this message? I feel it is former but just
> wanted to verify.
> >
> > Dhruv
>
Re: ICompletedTask semantics
Posted by Dhruv Mahajan <dh...@gmail.com>.
I feel if dispose fails, Task should still be termed as completed and
failure should be ignored. Can we think of scenarios in which this logic is
bad?
Dhruv
On Fri, Apr 8, 2016 at 11:18 AM, Andrew Chung <af...@gmail.com> wrote:
> Hi Dhruv,
>
> Interesting question. There might actually be a race condition here
> (somewhat, since I'm not sure if we count the `Dispose` call as part
> of the Task), given that `Dispose` is actually called *after* setting
> the Task status to `Completed`, and the Task thread locks on a
> separate object than the Heartbeat thread. Only the Task status set
> operation is atomic. Thus, if the Heartbeat thread gets the Task
> status and sends the message to the Driver prior to the Dispose is
> called, it is possible that Task.Dispose is not yet called before the
> Driver receives the Task completion message.
>
> Given that we haven't defined a particular order for this sequence of
> events, perhaps we should change the behavior to what Dhruv expects?
>
> Another thing to consider is whether we count `Dispose` as part of the
> Task. e.g. If Dispose fails, do we count the Task as failed (Somewhat
> dangerous, since many users don't implement Dispose on their Tasks)?
> Do we only count the Task as Completed after the Dispose is done? What
> do you think?
>
> Thanks,
> Andrew
>
> On Fri, Apr 8, 2016 at 10:26 AM, Julia Wang (QIUHE)
> <Qi...@microsoft.com> wrote:
> > It has been disposed I believe.
> >
> > -----Original Message-----
> > From: Dhruv Mahajan [mailto:dhruv.mahajan@gmail.com]
> > Sent: Friday, April 8, 2016 10:07 AM
> > To: dev@reef.apache.org
> > Subject: ICompletedTask semantics
> >
> > Hi
> >
> > I have a question regarding the semantics of ICompletedTask. Does this
> mean that Dispose on the Task has already been called or it will be called
> after driver acks. this message? I feel it is former but just wanted to
> verify.
> >
> > Dhruv
>
Re: ICompletedTask semantics
Posted by Andrew Chung <af...@gmail.com>.
Hi Dhruv,
Interesting question. There might actually be a race condition here
(somewhat, since I'm not sure if we count the `Dispose` call as part
of the Task), given that `Dispose` is actually called *after* setting
the Task status to `Completed`, and the Task thread locks on a
separate object than the Heartbeat thread. Only the Task status set
operation is atomic. Thus, if the Heartbeat thread gets the Task
status and sends the message to the Driver prior to the Dispose is
called, it is possible that Task.Dispose is not yet called before the
Driver receives the Task completion message.
Given that we haven't defined a particular order for this sequence of
events, perhaps we should change the behavior to what Dhruv expects?
Another thing to consider is whether we count `Dispose` as part of the
Task. e.g. If Dispose fails, do we count the Task as failed (Somewhat
dangerous, since many users don't implement Dispose on their Tasks)?
Do we only count the Task as Completed after the Dispose is done? What
do you think?
Thanks,
Andrew
On Fri, Apr 8, 2016 at 10:26 AM, Julia Wang (QIUHE)
<Qi...@microsoft.com> wrote:
> It has been disposed I believe.
>
> -----Original Message-----
> From: Dhruv Mahajan [mailto:dhruv.mahajan@gmail.com]
> Sent: Friday, April 8, 2016 10:07 AM
> To: dev@reef.apache.org
> Subject: ICompletedTask semantics
>
> Hi
>
> I have a question regarding the semantics of ICompletedTask. Does this mean that Dispose on the Task has already been called or it will be called after driver acks. this message? I feel it is former but just wanted to verify.
>
> Dhruv
RE: ICompletedTask semantics
Posted by "Julia Wang (QIUHE)" <Qi...@microsoft.com>.
It has been disposed I believe.
-----Original Message-----
From: Dhruv Mahajan [mailto:dhruv.mahajan@gmail.com]
Sent: Friday, April 8, 2016 10:07 AM
To: dev@reef.apache.org
Subject: ICompletedTask semantics
Hi
I have a question regarding the semantics of ICompletedTask. Does this mean that Dispose on the Task has already been called or it will be called after driver acks. this message? I feel it is former but just wanted to verify.
Dhruv