You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Stefan Fuhrmann <st...@wandisco.com> on 2012/11/25 23:17:38 UTC
Test code coverage
Hi there,
I was wondering whether we could have one or two of
our UNIX build bots create a code coverage profile
and make the results available online.
Here is what I use in my test runner script:
[[[
env CFLAGS='-fprofile-arcs -ftest-coverage' ./configure --disable-shared
--enable-maintainer-mode $moreopts
make -sj 2> /dev/null > /dev/null
make svnserveautocheck PARALLEL=1
lcov -d . -b . -c -o tests.lcov > lcov.log
genhtml tests.lcov -o html > genhtml.log
]]]
So, it's just 1 changed and 2 extra lines.
-- Stefan^2.
--
Certified & Supported Apache Subversion Downloads:
*
http://www.wandisco.com/subversion/download
*
Re: Test code coverage
Posted by Julian Foad <ju...@btopenworld.com>.
Stefan Fuhrmann wrote:
> Julian Foad wrote:
>> Hyrum K Wright wrote:
>>> Stefan Fuhrmann wrote:
>>>> I was wondering whether we could have one or two of
>>>> our UNIX build bots create a code coverage profile
>>>> and make the results available online.
[...]
>> I attach an updated patch. It uses 'lcov' instead of 'gcov'
>> as Stefan suggests, which makes the Makefile target much
>> simpler; and I modeled the 'configure' handling on the
>> existing --enable-gprof switch.
>
> Looks good.
>
>> The existing --enable-gprof option requires --disable-shared
>> and recommends --enable-all-static, and I have just copied that
>> without understanding why or whether the same applies to gcov.
>
> Given the bindings problem you ran into with an all-static build,
> I wonder whether it should rather be the other way around.
I have left it as it is, since
configure --enable-gcov --disable-shared --without-gnome-keyring
works for me if I avoid swig-py and swig rb bindings.
(With --enable-all-static Subversion doesn't build for me, but judging by the error messages that's probably because I don't have static versions of the dependencies installed.)
>> I'll probably commit it soon if there are no objections, even
>> though my testing has been rather limited, as it seems unlikely
>> to cause problems for anyone except those who use it.
>
> Since this is a developers only option, it is sufficient if we
> know about at least one working configuration.
True. Committed in r1416646.
- Julian
Re: Test code coverage
Posted by Stefan Fuhrmann <st...@wandisco.com>.
On Thu, Nov 29, 2012 at 12:14 AM, Julian Foad <ju...@btopenworld.com>wrote:
> Hyrum K Wright wrote:
>
> > On Sun, Nov 25, 2012 at 5:17 PM, Stefan Fuhrmann wrote:
> >> I was wondering whether we could have one or two of
> >> our UNIX build bots create a code coverage profile
> >> and make the results available online.
> >>
> >> Here is what I use in my test runner script:
> >>
> >> [[[
> >> env CFLAGS='-fprofile-arcs -ftest-coverage' ./configure
> --disable-shared --enable-maintainer-mode $moreopts
> >>
> >> make -sj 2> /dev/null > /dev/null
> >> make svnserveautocheck PARALLEL=1
> >>
> >> lcov -d . -b . -c -o tests.lcov > lcov.log
> >> genhtml tests.lcov -o html > genhtml.log
> >> ]]]
> >
> >
> > +1 to test coverage stats.
> >
> >
> > We have an ancient patch [...] in tools/dev/gcov.patch. [...]
>
> I attach an updated patch. It uses 'lcov' instead of 'gcov' as Stefan
> suggests, which makes the Makefile target much simpler; and I modeled the
> 'configure' handling on the existing --enable-gprof switch.
>
Looks good.
> The existing --enable-gprof option requires --disable-shared and
> recommends --enable-all-static, and I have just copied that without
> understanding why or whether the same applies to gcov.
>
Given the bindings problem you ran into with an all-static build,
I wonder whether it should rather be the other way around.
I'll probably commit it soon if there are no objections, even though my
> testing has been rather limited, as it seems unlikely to cause problems for
> anyone except those who use it.
>
Since this is a developers only option, it is sufficient if we
know about at least one working configuration.
-- Stefan^2.
--
Certified & Supported Apache Subversion Downloads:
*
http://www.wandisco.com/subversion/download
*
Re: Test code coverage
Posted by Julian Foad <ju...@btopenworld.com>.
Hyrum K Wright wrote:
> On Sun, Nov 25, 2012 at 5:17 PM, Stefan Fuhrmann wrote:
>> I was wondering whether we could have one or two of
>> our UNIX build bots create a code coverage profile
>> and make the results available online.
>>
>> Here is what I use in my test runner script:
>>
>> [[[
>> env CFLAGS='-fprofile-arcs -ftest-coverage' ./configure --disable-shared --enable-maintainer-mode $moreopts
>>
>> make -sj 2> /dev/null > /dev/null
>> make svnserveautocheck PARALLEL=1
>>
>> lcov -d . -b . -c -o tests.lcov > lcov.log
>> genhtml tests.lcov -o html > genhtml.log
>> ]]]
>
>
> +1 to test coverage stats.
>
>
> We have an ancient patch [...] in tools/dev/gcov.patch. [...]
I attach an updated patch. It uses 'lcov' instead of 'gcov' as Stefan suggests, which makes the Makefile target much simpler; and I modeled the 'configure' handling on the existing --enable-gprof switch.
The existing --enable-gprof option requires --disable-shared and recommends --enable-all-static, and I have just copied that without understanding why or whether the same applies to gcov.
I'll probably commit it soon if there are no objections, even though my testing has been rather limited, as it seems unlikely to cause problems for anyone except those who use it.
- Julian
Re: Test code coverage
Posted by Hyrum K Wright <hy...@hyrumwright.org>.
On Sun, Nov 25, 2012 at 5:17 PM, Stefan Fuhrmann <
stefan.fuhrmann@wandisco.com> wrote:
> Hi there,
>
> I was wondering whether we could have one or two of
> our UNIX build bots create a code coverage profile
> and make the results available online.
>
> Here is what I use in my test runner script:
>
> [[[
> env CFLAGS='-fprofile-arcs -ftest-coverage' ./configure --disable-shared
> --enable-maintainer-mode $moreopts
>
> make -sj 2> /dev/null > /dev/null
> make svnserveautocheck PARALLEL=1
>
> lcov -d . -b . -c -o tests.lcov > lcov.log
> genhtml tests.lcov -o html > genhtml.log
> ]]]
+1 to test coverage stats.
We have an ancient patch in our repo for making Subversion build and use
gcov to produce coverage information. I think it's in
tools/dev/gcov.patch. I don't know if it's useful or not, but as you're
looking at this topic, it might be a good time to revisit whether we need
it there or not.
-Hyrum
Re: Test code coverage
Posted by Ben Reser <br...@gmail.com>.
On Nov 28, 2012 1:55 PM, "Julian Foad" <ju...@btopenworld.com> wrote:
>
> I tried this on the Centos buildbot today and ran into another issue: the
Perl bindings build fails at link stage. See <
http://ci.apache.org/builders/svn-x64-centos-gcc/builds/7765/steps/Build/logs/stdio>
or here are the last few lines:
> [[[
> Running Mkbootstrap for SVN::_Core ()
> chmod 644 _Core.bs
> rm -f blib/arch/auto/SVN/_Core/_Core.so
> gcc -shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic core.o -o
blib/arch/auto/SVN/_Core/_Core.so -L[...] -l[...]
> /usr/bin/ld:
/home/bt/slaves/x64-centos/build/subversion/libsvn_diff/.libs/libsvn_diff-1.a(deprecated.o):
relocation R_X86_64_32 against `a local symbol' can not be used when making
a shared object; recompile with -fPIC
> ]]]
>
> It looks like the effect of --disable-shared and the two '-f' flags
aren't getting passed along to this link stage.
I'll see what I can do. Perl bindings use the CFLAGs from the perl
install, not ours build system settings.
Re: Test code coverage
Posted by Julian Foad <ju...@btopenworld.com>.
Stefan Fuhrmann wrote:
> Julian Foad wrote:
>>> [[[
>>> env CFLAGS='-fprofile-arcs -ftest-coverage' ./configure \
>>> --disable-shared --enable-maintainer-mode $moreopts
>>>
>>> make -sj 2> /dev/null > /dev/null
I tried this on the Centos buildbot today and ran into another issue: the Perl bindings build fails at link stage. See <http://ci.apache.org/builders/svn-x64-centos-gcc/builds/7765/steps/Build/logs/stdio> or here are the last few lines:
[[[
Running Mkbootstrap for SVN::_Core ()
chmod 644 _Core.bs
rm -f blib/arch/auto/SVN/_Core/_Core.so
gcc -shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic core.o -o blib/arch/auto/SVN/_Core/_Core.so -L[...] -l[...]
/usr/bin/ld: /home/bt/slaves/x64-centos/build/subversion/libsvn_diff/.libs/libsvn_diff-1.a(deprecated.o): relocation R_X86_64_32 against `a local symbol' can not be used when making a shared object; recompile with -fPIC
]]]
It looks like the effect of --disable-shared and the two '-f' flags aren't getting passed along to this link stage.
>>> make svnserveautocheck PARALLEL=1
>>
>> Some of the Python tests error out when I run them in parallel
>> like this. The errors are:
>> [[[
>> W: profiling:/.../subversion/libsvn_subr/.libs/nls.gcda:Invocation mismatch - some data files may have been removed
>> ]]]
>
> Hm. It works for me (except for 1 test that fails from time to time).
> I run that test script only as a quick check for major breakage.
>
>
>> I found an email <http://gcc.gnu.org/ml/gcc-help/2005-11/msg00074.html>
>> which says "gcov instrumentation is multi-process safe, but not
>> multi-thread safe. The multi-processing safety relies on OS level file
>> locking, which is not available on some systems.".
>>
>>> lcov -d . -b . -c -o tests.lcov > lcov.log
>>> genhtml tests.lcov -o html > genhtml.log
>>> ]]]
>>
>> OK, that part works.
>>
>> Is there a command to reset the coverage statistics, that I can
>> do before a test run if I don't want to 'make clean'?
>
> Not sure 'make clean' would even help. This is what I do:
'make clean' does work because it deletes all the '.libs' directories, and the coverage data files were created in those dirs.
> find . -name "*.gc*" -print0 | xargs -0 rm -rf
Thanks.
- Julian
Re: Test code coverage
Posted by Stefan Fuhrmann <st...@wandisco.com>.
On Mon, Nov 26, 2012 at 7:56 PM, Julian Foad <ju...@btopenworld.com>wrote:
> Stefan Fuhrmann wrote:
>
> > I was wondering whether we could have one or two of
> > our UNIX build bots create a code coverage profile
> > and make the results available online.
> >
> > Here is what I use in my test runner script:
> >
> > [[[
> > env CFLAGS='-fprofile-arcs -ftest-coverage' ./configure \
> > --disable-shared --enable-maintainer-mode $moreopts
> >
> > make -sj 2> /dev/null > /dev/null
> > make svnserveautocheck PARALLEL=1
>
>
> Some of the Python tests error out when I run them in parallel like this.
> The errors are:
> [[[
> W: profiling:/.../subversion/libsvn_subr/.libs/nls.gcda:Invocation
> mismatch - some data files may have been removed
> ]]]
>
Hm. It works for me (except for 1 test that fails from time to time).
I run that test script only as a quick check for major breakage.
> I found an email <http://gcc.gnu.org/ml/gcc-help/2005-11/msg00074.html>
> which says "gcov instrumentation is multi-process safe, but not
> multi-thread safe. The
> multi-processing safety relies on OS level file locking, which is not
> available
> on some systems.".
>
> >
> > lcov -d . -b . -c -o tests.lcov > lcov.log
> > genhtml tests.lcov -o html > genhtml.log
> > ]]]
>
>
> OK, that part works.
>
> Is there a command to reset the coverage statistics, that I can do before
> a test run if I don't want to 'make clean'?
>
Not sure 'make clean' would even help. This is what I do:
find . -name "*.gc*" -print0 | xargs -0 rm -rf
-- Stefan^2.
--
Certified & Supported Apache Subversion Downloads:
*
http://www.wandisco.com/subversion/download
*
Re: Test code coverage
Posted by Branko Čibej <br...@wandisco.com>.
On 26.11.2012 19:56, Julian Foad wrote:
> Stefan Fuhrmann wrote:
>
>> I was wondering whether we could have one or two of
>> our UNIX build bots create a code coverage profile
>> and make the results available online.
>>
>> Here is what I use in my test runner script:
>>
>> [[[
>> env CFLAGS='-fprofile-arcs -ftest-coverage' ./configure \
>> --disable-shared --enable-maintainer-mode $moreopts
>>
>> make -sj 2> /dev/null > /dev/null
>> make svnserveautocheck PARALLEL=1
>
> Some of the Python tests error out when I run them in parallel like this. The errors are:
> [[[
> W: profiling:/.../subversion/libsvn_subr/.libs/nls.gcda:Invocation mismatch - some data files may have been removed
> ]]]
>
> I found an email <http://gcc.gnu.org/ml/gcc-help/2005-11/msg00074.html> which says "gcov instrumentation is multi-process safe, but not multi-thread safe. The
> multi-processing safety relies on OS level file locking, which is not available
> on some systems.".
Maybe we should replace threading with multiprocess to run parallel
python tests. They'd block less in the GIL as well.
-- Brane
--
Branko Čibej
Director of Subversion | WANdisco | www.wandisco.com
Re: Test code coverage
Posted by Julian Foad <ju...@btopenworld.com>.
Stefan Fuhrmann wrote:
> I was wondering whether we could have one or two of
> our UNIX build bots create a code coverage profile
> and make the results available online.
>
> Here is what I use in my test runner script:
>
> [[[
> env CFLAGS='-fprofile-arcs -ftest-coverage' ./configure \
> --disable-shared --enable-maintainer-mode $moreopts
>
> make -sj 2> /dev/null > /dev/null
> make svnserveautocheck PARALLEL=1
Some of the Python tests error out when I run them in parallel like this. The errors are:
[[[
W: profiling:/.../subversion/libsvn_subr/.libs/nls.gcda:Invocation mismatch - some data files may have been removed
]]]
I found an email <http://gcc.gnu.org/ml/gcc-help/2005-11/msg00074.html> which says "gcov instrumentation is multi-process safe, but not multi-thread safe. The
multi-processing safety relies on OS level file locking, which is not available
on some systems.".
>
> lcov -d . -b . -c -o tests.lcov > lcov.log
> genhtml tests.lcov -o html > genhtml.log
> ]]]
OK, that part works.
Is there a command to reset the coverage statistics, that I can do before a test run if I don't want to 'make clean'?
- Julian