Unit test for actual REEF job in local run-time

[Dhruv Mahajan <dh...@gmail.com>]

Hi

How do we write unit tests for actual REEF jobs in local runtime. Is there
any formal mechanism? I knew something existed on .Net side but then with
the introduction of proper REEF client is the mechanism still valid?
Basically I am thinking of printing some Info messages in the logs and then
parsing the logs to see if the output is expected.

Of course the tricky part is consistency. What if somebody removes the Log
message. The the test will not be valid anymore.

One more ambitious way and probably we have to see the feasibility of it is
to consider whether we can track certain variables in REEF or introduce
another Level of logging say Level.Test.

Dhruv

Re: Unit test for actual REEF job in local run-time

[Markus Weimer <ma...@weimo.de>]

On 2015-09-17 09:19, Dhruv Mahajan wrote:
> How do we write unit tests for actual REEF jobs in local runtime. Is there
> any formal mechanism?

We have a mechanism in Java, and all the tests in `reef-tests` use it:

The unit test framework runs the Client side of the job, and on the
Driver side, we throw specific exceptions like `DriverSideFailure` or,
on the Evaluator, `EvaluatorSideFailure`. We then rely on REEF's
exception captioning and forwarding feature to get theses to the client,
where the unit test passes or fails. Of course, there are tests that
test the exception forwarding itself :-)

The tests get their runtime configuration via the
`org.apache.reef.tests.TestEnvironment` mechanism. This makes the tests
run equally on YARN and the local runtime.

We don't have a full client for REEF.NET yet, so the same mechanism
can't be applied. There are several JIRAs with the goal of providing
such a client:

https://issues.apache.org/jira/browse/REEF-189
https://issues.apache.org/jira/browse/REEF-70
https://issues.apache.org/jira/browse/REEF-663

I believe Anupam is actively looking into those and can comment on the
state of things.

> Basically I am thinking of printing some Info messages in the logs and then
> parsing the logs to see if the output is expected.

That is what is being done today, yes. Writing tests this way is better
than having none. But clearly, they need to be refactored once we have a
client :-)

Markus