You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Dhruv Mahajan <dh...@gmail.com> on 2016/07/01 02:35:14 UTC
IMRU driver does not exist even after all tasks complete
Hi
Intermittently I am finding this issue that IMRU driver does not exit even
if all tasks are completed. I looked for
messages and it seems the count is exactly equal to the total tasks. I am
not sure it is a general REEF issue now or IMRU.
Re: IMRU driver does not exist even after all tasks complete
Posted by Andrew Chung <af...@gmail.com>.
Hi Dhruv,
The problem with the previous code in the Driver was that when
checking whether or not an Evaluator has ended, it checks its event
queue for events that are still in the queue. If there are still
events in the queue, the EvaluatorManager will return not-idle to the
driver and thus preventing shutdown. However, the previous code did
not consider events that are still being processed by an EventHandler.
The PR fixed it by incrementing a counter when entering an
EventHandler and decrementing the counter when exiting it (see
`ThreadPoolStage` in the PR, the relevant calls are `beforeOnNext()`
and `afterOnNext()`), complete with a Thread that checks the counter
repeatedly until it's declared completed when an Evaluator is shut
down. `EvaluatorIdlenessThreadPoolSize is the size of the thread pool
for checking the counter on all `EvaluatorManagers`. This seems like
the most suspicious code which may cause the driver to fail to shut
down after all Evaluators have completed. It might be useful to add
logging statements with a higher error level upon checking for
Evaluator completion to see if this is really the problem.
Thanks,
Andrew
On Sun, Jul 3, 2016 at 9:31 AM, Dhruv Mahajan <dh...@gmail.com> wrote:
> you mean this parameter: EvaluatorIdlenessThreadPoolSize. Can you explain
> what this part of the code does?
>
> On Fri, Jul 1, 2016 at 5:09 PM, Andrew Chung <af...@gmail.com> wrote:
>
>> Hi Dhruv,
>>
>> When has this begun to happen?
>>
>> I suspect it might be this PR:
>>
>> https://github.com/apache/reef/pull/1007
>>
>> that caused the problem. You may want to modify the logs a bit or try
>> increasing the thread count there to see if this still occurs.
>>
>> Thanks,
>> Andrew
>>
>> On Fri, Jul 1, 2016 at 4:38 PM, Dhruv Mahajan <dh...@gmail.com>
>> wrote:
>> > I ran IMRU for 4-5 more times and again got this issue once. I feel there
>> > is some race condition in base REEF code.
>> >
>> > On Fri, Jul 1, 2016 at 11:25 AM, Dhruv Mahajan <dh...@gmail.com>
>> > wrote:
>> >
>> >> I got all the completed evaluators messages. SO I guess they were right?
>> >>
>> >> Dhruv
>> >>
>> >> On Fri, Jul 1, 2016 at 8:43 AM, Andrew Chung <af...@gmail.com>
>> wrote:
>> >>
>> >>> Were the Contexts/Evaluators disposed upon Task completion in the IMRU
>> >>> driver?
>> >>>
>> >>> Thanks,
>> >>> Andrew
>> >>>
>> >>> On Thu, Jun 30, 2016 at 7:35 PM, Dhruv Mahajan <
>> dhruv.mahajan@gmail.com>
>> >>> wrote:
>> >>> > Hi
>> >>> >
>> >>> > Intermittently I am finding this issue that IMRU driver does not exit
>> >>> even
>> >>> > if all tasks are completed. I looked for
>> >>> >
>> >>> > messages and it seems the count is exactly equal to the total tasks.
>> I
>> >>> am
>> >>> > not sure it is a general REEF issue now or IMRU.
>> >>>
>> >>
>> >>
>>
Re: IMRU driver does not exist even after all tasks complete
Posted by Dhruv Mahajan <dh...@gmail.com>.
you mean this parameter: EvaluatorIdlenessThreadPoolSize. Can you explain
what this part of the code does?
On Fri, Jul 1, 2016 at 5:09 PM, Andrew Chung <af...@gmail.com> wrote:
> Hi Dhruv,
>
> When has this begun to happen?
>
> I suspect it might be this PR:
>
> https://github.com/apache/reef/pull/1007
>
> that caused the problem. You may want to modify the logs a bit or try
> increasing the thread count there to see if this still occurs.
>
> Thanks,
> Andrew
>
> On Fri, Jul 1, 2016 at 4:38 PM, Dhruv Mahajan <dh...@gmail.com>
> wrote:
> > I ran IMRU for 4-5 more times and again got this issue once. I feel there
> > is some race condition in base REEF code.
> >
> > On Fri, Jul 1, 2016 at 11:25 AM, Dhruv Mahajan <dh...@gmail.com>
> > wrote:
> >
> >> I got all the completed evaluators messages. SO I guess they were right?
> >>
> >> Dhruv
> >>
> >> On Fri, Jul 1, 2016 at 8:43 AM, Andrew Chung <af...@gmail.com>
> wrote:
> >>
> >>> Were the Contexts/Evaluators disposed upon Task completion in the IMRU
> >>> driver?
> >>>
> >>> Thanks,
> >>> Andrew
> >>>
> >>> On Thu, Jun 30, 2016 at 7:35 PM, Dhruv Mahajan <
> dhruv.mahajan@gmail.com>
> >>> wrote:
> >>> > Hi
> >>> >
> >>> > Intermittently I am finding this issue that IMRU driver does not exit
> >>> even
> >>> > if all tasks are completed. I looked for
> >>> >
> >>> > messages and it seems the count is exactly equal to the total tasks.
> I
> >>> am
> >>> > not sure it is a general REEF issue now or IMRU.
> >>>
> >>
> >>
>
Re: IMRU driver does not exist even after all tasks complete
Posted by Andrew Chung <af...@gmail.com>.
Hi Dhruv,
When has this begun to happen?
I suspect it might be this PR:
https://github.com/apache/reef/pull/1007
that caused the problem. You may want to modify the logs a bit or try
increasing the thread count there to see if this still occurs.
Thanks,
Andrew
On Fri, Jul 1, 2016 at 4:38 PM, Dhruv Mahajan <dh...@gmail.com> wrote:
> I ran IMRU for 4-5 more times and again got this issue once. I feel there
> is some race condition in base REEF code.
>
> On Fri, Jul 1, 2016 at 11:25 AM, Dhruv Mahajan <dh...@gmail.com>
> wrote:
>
>> I got all the completed evaluators messages. SO I guess they were right?
>>
>> Dhruv
>>
>> On Fri, Jul 1, 2016 at 8:43 AM, Andrew Chung <af...@gmail.com> wrote:
>>
>>> Were the Contexts/Evaluators disposed upon Task completion in the IMRU
>>> driver?
>>>
>>> Thanks,
>>> Andrew
>>>
>>> On Thu, Jun 30, 2016 at 7:35 PM, Dhruv Mahajan <dh...@gmail.com>
>>> wrote:
>>> > Hi
>>> >
>>> > Intermittently I am finding this issue that IMRU driver does not exit
>>> even
>>> > if all tasks are completed. I looked for
>>> >
>>> > messages and it seems the count is exactly equal to the total tasks. I
>>> am
>>> > not sure it is a general REEF issue now or IMRU.
>>>
>>
>>
Re: IMRU driver does not exist even after all tasks complete
Posted by Dhruv Mahajan <dh...@gmail.com>.
I ran IMRU for 4-5 more times and again got this issue once. I feel there
is some race condition in base REEF code.
On Fri, Jul 1, 2016 at 11:25 AM, Dhruv Mahajan <dh...@gmail.com>
wrote:
> I got all the completed evaluators messages. SO I guess they were right?
>
> Dhruv
>
> On Fri, Jul 1, 2016 at 8:43 AM, Andrew Chung <af...@gmail.com> wrote:
>
>> Were the Contexts/Evaluators disposed upon Task completion in the IMRU
>> driver?
>>
>> Thanks,
>> Andrew
>>
>> On Thu, Jun 30, 2016 at 7:35 PM, Dhruv Mahajan <dh...@gmail.com>
>> wrote:
>> > Hi
>> >
>> > Intermittently I am finding this issue that IMRU driver does not exit
>> even
>> > if all tasks are completed. I looked for
>> >
>> > messages and it seems the count is exactly equal to the total tasks. I
>> am
>> > not sure it is a general REEF issue now or IMRU.
>>
>
>
Re: IMRU driver does not exist even after all tasks complete
Posted by Dhruv Mahajan <dh...@gmail.com>.
I got all the completed evaluators messages. SO I guess they were right?
Dhruv
On Fri, Jul 1, 2016 at 8:43 AM, Andrew Chung <af...@gmail.com> wrote:
> Were the Contexts/Evaluators disposed upon Task completion in the IMRU
> driver?
>
> Thanks,
> Andrew
>
> On Thu, Jun 30, 2016 at 7:35 PM, Dhruv Mahajan <dh...@gmail.com>
> wrote:
> > Hi
> >
> > Intermittently I am finding this issue that IMRU driver does not exit
> even
> > if all tasks are completed. I looked for
> >
> > messages and it seems the count is exactly equal to the total tasks. I am
> > not sure it is a general REEF issue now or IMRU.
>
Re: IMRU driver does not exist even after all tasks complete
Posted by Andrew Chung <af...@gmail.com>.
Were the Contexts/Evaluators disposed upon Task completion in the IMRU driver?
Thanks,
Andrew
On Thu, Jun 30, 2016 at 7:35 PM, Dhruv Mahajan <dh...@gmail.com> wrote:
> Hi
>
> Intermittently I am finding this issue that IMRU driver does not exit even
> if all tasks are completed. I looked for
>
> messages and it seems the count is exactly equal to the total tasks. I am
> not sure it is a general REEF issue now or IMRU.