You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@reef.apache.org by Mariia Mykhailova <ma...@microsoft.com> on 2016/06/15 17:19:56 UTC
RE: Error messages in reef functional tests
For the "program has stopped working" error pop-ups, you can try following instructions from
https://www.raymond.cc/blog/disable-program-has-stopped-working-error-dialog-in-windows-server-2008/
Enabling "prevent display of the user interface for critical errors" seems to have helped on Julia's machine.
-Mariia
-----Original Message-----
From: Julia Wang (QIUHE) [mailto:Qiuhe.Wang@microsoft.com]
Sent: Tuesday, May 24, 2016 3:13 PM
To: dev@reef.apache.org
Subject: RE: Error messages in reef functional tests
Looks like the screen shot becomes text in my previous email. Basically I got the same message box in .Net testing that says:
messages -REEF.Bridge.exe has stopped
Julia
-----Original Message-----
From: Dhruv Mahajan [mailto:dhruv.mahajan@gmail.com]
Sent: Monday, May 16, 2016 3:41 PM
To: dev@reef.apache.org
Subject: Re: Error messages in reef functional tests
Also I cloned reef at some other location on my machine and re-ran tests.
Got the same errors in tests. This rules out cleaning issues.
Dhruv
On Mon, May 16, 2016 at 3:38 PM, Andrew Chung <af...@gmail.com> wrote:
> I've gotten the test logs from Dhruv, and judging from what I see in
> the logs prior to looking into the code, what is happening seems like
> the
> following:
>
> 1. Evaluator fails.
> 2. FailedEvaluatorHandler gets called.
> 3. We send request for another Evaluator in FailedEvaluatorHandler.
> 4. Before the EvaluatorRequestor sends the request, the Driver
> idleness handler kicks in and detects that the Driver is idle, i.e. no
> active Evaluators and no outstanding requests.
> 5. Driver is declared as idle and shut down.
> 6. EvaluatorRequestor processes requests. Evaluator starts as Driver
> is shutting down.
> 7. Test failure.
>
> I'll take a closer look at the code and provide updates.
>
> Thanks,
> Andrew
>
> On Mon, May 16, 2016 at 3:03 PM, Dhruv Mahajan
> <dh...@gmail.com>
> wrote:
>
> > Did deep cleaning and ran tests from command line...Still getting
> > one error. Also got the error window.
> >
> >
> >
> Org.Apache.REEF.Tests.Functional.Bridge.TestUnhandledTaskException.Tes
> tUnhandledTaskExceptionCrashesEvaluator
> >
> > Expected number of evaluators to fail (2) differs from actual number
> > of failed evaluator indicators (1)\r\nExpected: True\r\nActual:
> > False
> >
> > at
> >
> Org.Apache.REEF.Tests.Functional.ReefFunctionalTest.ValidateSuccessFor
> LocalRuntime(Int32
> > numberOfContextsToClose, Int32 numberOfTasksToFail, Int32
> > numberOfEvaluatorsToFail, String testFolder) in
> >
> >
> c:\Users\dhrumaha\Desktop\projects\reef\lang\cs\Org.Apache.REEF.Tests\
> Functional\ReefFunctionalTest.cs:line
> > 169
> > at
> >
> Org.Apache.REEF.Tests.Functional.Bridge.TestUnhandledTaskException.Tes
> tUnhandledTaskExceptionCrashesEvaluator()
> > in
> >
> c:\Users\dhrumaha\Desktop\projects\reef\lang\cs\Org.Apache.REEF.Tests\
> Functional\Bridge\TestUnhandledTaskException.cs:line
> > 54
> >
> >
> > On Mon, May 16, 2016 at 2:29 PM, Dhruv Mahajan
> > <dh...@gmail.com>
> > wrote:
> >
> > > Command line ... Lemme do deep clean once again.
> > >
> > > Dhruv
> > >
> > > On Mon, May 16, 2016 at 2:27 PM, Markus Weimer <ma...@weimo.de>
> wrote:
> > >
> > >> Are you running from the command line or Visual Studio? The tests
> > >> pass
> > all
> > >> the time from the command line for me, and AppVeyor now reports
> > >> the
> same
> > >> ...
> > >>
> > >> Markus
> > >>
> > >
> > >
> >
>
RE: Error messages in reef functional tests
Posted by Mariia Mykhailova <ma...@microsoft.com>.
Yes, I'll add this to build instructions once I have second confirmation from somebody for whom this fix helped remove these boxes. I don't see these boxes at all, so I can't test the fix on my machine.
-Mariia
-----Original Message-----
From: Markus Weimer [mailto:markus@weimo.de]
Sent: Thursday, June 16, 2016 7:52 AM
To: dev@reef.apache.org
Subject: Re: Error messages in reef functional tests
On 2016-06-15 10:46, Mariia Mykhailova wrote:
> I believe they are triggered by tests which crash evaluator.
Ok. In that case, we should document this as part of our build instructions. This can easily alarm people who build & test REEF for the first time. We had the following text in the Java build instructions for some time now because of a similar issue:
> Note that the tests will take several minutes to complete. You will
> also see stack traces fly by. Not to worry: those are part of the
> tests that test REEF's error reporting.
Markus
Re: Error messages in reef functional tests
Posted by Markus Weimer <ma...@weimo.de>.
On 2016-06-16 08:04, Dhruv Mahajan wrote:
> Another long term option is to replace these functional tests by proper
> exception throwing or logging mechanism where these logs or messages are
> actually transferred from driver to client.
I am not sure this would remove the dialog boxes. If I understand
correctly, they are generated whenever a process exits with anything
other than `0` as the return code.
When an Evaluator fails with an exception, I believe we should not
`return 0` in its `main` function, as e.g. the YARN NodeManager relies
on these error codes for its bookkeeping.
However, if we successfully transferred an exception to the upstream
process (the Driver for an Evaluator and the Client for the Driver), we
could consider this an error-free exit of the Evaluator process itself.
I'm not very familiar with the best practices when it comes to return
code (other than `0` indicating success). Are any of you? It would be
good for REEF processes to be well behaved citizens on Windows and Linux :)
Markus
Re: Error messages in reef functional tests
Posted by Dhruv Mahajan <dh...@gmail.com>.
Another long term option is to replace these functional tests by proper
exception throwing or logging mechanism where these logs or messages are
actually transferred from driver to client. Tasks can first send them to
driver which can report them back to client.
Dhruv
On Jun 16, 2016 7:52 AM, "Markus Weimer" <ma...@weimo.de> wrote:
On 2016-06-15 10:46, Mariia Mykhailova wrote:
> I believe they are triggered by tests which crash evaluator.
>
Ok. In that case, we should document this as part of our build
instructions. This can easily alarm people who build & test REEF for the
first time. We had the following text in the Java build instructions for
some time now because of a similar issue:
Note that the tests will take several minutes to complete. You will
> also see stack traces fly by. Not to worry: those are part of the
> tests that test REEF's error reporting.
>
Markus
Re: Error messages in reef functional tests
Posted by Markus Weimer <ma...@weimo.de>.
On 2016-06-15 10:46, Mariia Mykhailova wrote:
> I believe they are triggered by tests which crash evaluator.
Ok. In that case, we should document this as part of our build
instructions. This can easily alarm people who build & test REEF for the
first time. We had the following text in the Java build instructions for
some time now because of a similar issue:
> Note that the tests will take several minutes to complete. You will
> also see stack traces fly by. Not to worry: those are part of the
> tests that test REEF's error reporting.
Markus
RE: Error messages in reef functional tests
Posted by Mariia Mykhailova <ma...@microsoft.com>.
I believe they are triggered by tests which crash evaluator. These boxes report (expected) crash and prevent the crashed process from closing until user closes the box, which in turn prevents evaluator logs from being written. If the box is kept open for longer than our per-test timeout of 60 sec, the test will not be able to access the log and will fail with "Cannot read log" error.
-Mariia
-----Original Message-----
From: Markus Weimer [mailto:markus@weimo.de]
Sent: Wednesday, June 15, 2016 10:38 AM
To: dev@reef.apache.org
Subject: Re: Error messages in reef functional tests
On 2016-06-15 10:19, Mariia Mykhailova wrote:
> Enabling "prevent display of the user interface for critical errors"
> seems to have helped on Julia's machine.
Isn't this hiding the problem rather than fixing it? I mean, it makes our dev lives easier, which is a win :)
Or are these dialog boxes triggered in the tests that actively fail the Evaluator? In that case, those Dialog boxes are indeed expected (and should be suppressed).
Markus
Re: Error messages in reef functional tests
Posted by Markus Weimer <ma...@weimo.de>.
On 2016-06-15 10:19, Mariia Mykhailova wrote:
> Enabling "prevent display of the user interface for critical errors"
> seems to have helped on Julia's machine.
Isn't this hiding the problem rather than fixing it? I mean, it makes
our dev lives easier, which is a win :)
Or are these dialog boxes triggered in the tests that actively fail the
Evaluator? In that case, those Dialog boxes are indeed expected (and
should be suppressed).
Markus