[DISCUSS] Add code coverage analysis

[Moaz Reyad <mo...@apache.org>]

Hi all,

Here is the second proposal for improving SINGA after the code quality
proposal [1]. This discussion is about adding code coverage to Travis CI
[2].

To enable code coverage, I think we need to fix some stuff first:

1. We need to make the Travis build fails if the test cases fail.
Currently, the test cases fail (as can be seen in the log, for example [3])
but the Travis build does not show this problem and finish with "Done. Your
build exited with 0." [4].

2. Some unit tests fail because they run on the wrong architecture, not
because there is a problem in the code. So we need to improve the test
design. For example, in [5] there is an error "AssertionError: SINGA has
not been compiled with CUDA enabled.". I think it is better to run CUDA
tests only if the code is compiled for CUDA. This should be the same for
all other compilation options and their corresponding test cases.

3. The python tests should run in python machines in Travis not in C++
machines [6]. Currently, the test cases uses C++ machines and install
python (with conda) on them to run the python tests, instead of running
python machines.

4. The python test for 3.6 [7] and 3.7 [8] should be done on different
machines in Travis where the native python packages of the operating system
(e.g. ubuntu or Mac) can be used. Currently, both tests happen on one C++
machine and conda is used instead of the python packages of the operating
system. This only tests SINGA build with conda, but not with python alone.

5. We need to run the C++ tests [9]. Currently only the python tests are
executed in Travis.

After these five issues are fixed, we can proceed to

6. Add C++ coverage
7. Add python coverage
8. Show the coverage results
9. Improve the coverage (i.e. add more C++ and python unit tests to cover
the code which is not tested)

If everyone agrees, I will proceed by creating sub tasks for these issues
under SINGA-485. Then we can implement them one by one.

Thanks,
Moaz

[1]
http://mail-archives.apache.org/mod_mbox/singa-dev/201909.mbox/%3CCAH%3DWUUbgvL62dZOCRJeEDA-a-JWNEyGvvK1Dn3ZL1YDvLXWktQ%40mail.gmail.com%3E
[2] https://issues.apache.org/jira/projects/SINGA/issues/SINGA-485
[3] https://travis-ci.org/apache/singa/jobs/613933817#L3702
[4] https://travis-ci.org/apache/singa/jobs/613933817#L3849
[5] https://travis-ci.org/apache/singa/jobs/613933817#L2879
[6] https://github.com/apache/singa/blob/master/.travis.yml#L19
[7] https://github.com/apache/singa/blob/master/tool/travis/build.sh#L33
[8] https://github.com/apache/singa/blob/master/tool/travis/build.sh#L34
[9] https://github.com/apache/singa/tree/master/test/singa