You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@madlib.apache.org by Ed Espino <es...@apache.org> on 2017/07/14 17:00:32 UTC
MADlib Code coverage
Out of curiosity, do MADlib developers regularly use code coverage
utilities to measure the coverage quality of their tests?
-=e
--
*Ed Espino*
Re: MADlib Code coverage
Posted by Nandish Jayaram <nj...@pivotal.io>.
This is great Ed, thank you. Will review the PR soon. This will be very helpful to MADlib developers.
NJ
Sent from my iPhone
> On Jul 14, 2017, at 11:14 PM, Ed Espino <es...@apache.org> wrote:
>
> Nandish,
>
> For a very basic code coverage implementation using cmake, I have created:
>
> * JIRA: https://issues.apache.org/jira/browse/MADLIB-1138
> * PR: https://github.com/apache/incubator-madlib/pull/151
>
> This is a basic code coverage implementation runs on my mac using the
> following and a brew installed lcov module:
>
> Configured with: --prefix=/Library/Developer/CommandLineTools/usr
> --with-gxx-include-dir=/usr/include/c++/4.2.1
> Apple LLVM version 8.1.0 (clang-802.0.42)
> Target: x86_64-apple-darwin16.6.0
> Thread model: posix
> InstalledDir: /Library/Developer/CommandLineTools/usr/bin
>
> Regards,
> -=e
>
>> On Fri, Jul 14, 2017 at 1:50 PM, Ed Espino <es...@apache.org> wrote:
>>
>> Sure NJ,
>>
>> I'll create both (Jira and PR).
>>
>> -=e
>>
>> On Fri, Jul 14, 2017 at 1:46 PM, Nandish Jayaram <nj...@pivotal.io>
>> wrote:
>>
>>> Thank you Ed, this is great. We will create a JIRA to incorporate this
>>> (including documenting the steps to use it). Or, you can also create a PR.
>>> :)
>>>
>>> NJ
>>>
>>>> On Fri, Jul 14, 2017 at 12:31 PM, Ed Espino <es...@apache.org> wrote:
>>>>
>>>> NJ,
>>>>
>>>> I spent about an hour getting it setup on my mac. This supported the
>>>> MLP PR review. I threw some info together below.
>>>>
>>>> Hope it helps,
>>>> -=e
>>>>
>>>> ------------------
>>>>
>>>> Add lcov through brew (brings in lcov and genhtml utilities):
>>>> brew install lcov
>>>>
>>>> "Total HACK" to add code coverage support into the build for C/C++
>>>> code. This is quick and dirty so I could get to the data I needed for
>>>> the PR review.
>>>>
>>>> diff --git a/CMakeLists.txt b/CMakeLists.txt
>>>> index b2172ef3..db80986c 100644
>>>> --- a/CMakeLists.txt
>>>> +++ b/CMakeLists.txt
>>>> @@ -104,7 +104,7 @@ if(CMAKE_COMPILER_IS_GNUCXX)
>>>> set(CMAKE_INCLUDE_SYSTEM_FLAG_CXX "-isystem ")
>>>> endif(APPLE)
>>>> elseif(CMAKE_C_COMPILER_ID MATCHES "Clang")
>>>> - set(CMAKE_CXX_FLAGS "-stdlib=libstdc++")
>>>> + set(CMAKE_CXX_FLAGS "-stdlib=libstdc++ -fprofile-arcs
>>>> -ftest-coverage")
>>>> endif(CMAKE_COMPILER_IS_GNUCXX)
>>>>
>>>> # force a `m4_' prefix to all builtins
>>>> @@ -114,6 +114,8 @@ else()
>>>> set(M4_ARGUMENTS "--prefix-builtins")
>>>> endif()
>>>>
>>>> +set(CMAKE_C_FLAGS "-fprofile-arcs -ftest-coverage")
>>>> +
>>>> # Read and parse Version.yml file
>>>> file(READ "${MADLIB_VERSION_YML}" _MADLIB_VERSION_CONTENTS)
>>>> string(REGEX REPLACE "^.*version:[ \t]*([^\n]*)\n.*" "\\1"
>>>> MADLIB_VERSION_STRING "${_MADLIB_VERSION_CONTENTS}")
>>>>
>>>> Once MADlib is built and installed, run your tests:
>>>>
>>>> ## Build and install MADlib
>>>> mkdir build
>>>> cd build
>>>> cmake ..
>>>> make -j8 install
>>>> /usr/local/madlib/bin/madpack -s madlib -p postgres install
>>>>
>>>> ## At this point you can run your tests (I'm focusing on MLP)
>>>>
>>>> /usr/local/madlib/bin/madpack -s madlib -p postgres install-check -t
>>>> convex/mlp
>>>>
>>>> ## Time to capture the results and generate html report.
>>>>
>>>> lcov --capture --directory . --output-file coverage.info
>>>> genhtml coverage.info --output-directory gcov
>>>>
>>>> # You will notice there is some uninteresting system coverage info.
>>>> # The following will filter them out.
>>>>
>>>> lcov --remove coverage.info '/usr/include/*' '/usr/local/include/*'
>>>> '/usr/local/postgres/*' '*build/third_party/*' -o
>>> coverage_filtered.info
>>>> rm -rf gcov
>>>> genhtml coverage_filtered.info --output-directory gcov
>>>>
>>>> It is possible to zero out the counters with the following.
>>>> lcov --zerocounters --directory .
>>>>
>>>>
>>>> On Fri, Jul 14, 2017 at 12:23 PM, Nandish Jayaram <nj...@pivotal.io>
>>>> wrote:
>>>>
>>>>> Hi Ed,
>>>>>
>>>>> We haven't set that up for MADlib yet, but we will be looking into it
>>>> soon.
>>>>> Any ideas?
>>>>>
>>>>> NJ
>>>>>
>>>>> On Fri, Jul 14, 2017 at 10:00 AM, Ed Espino <es...@apache.org>
>>> wrote:
>>>>>
>>>>>> Out of curiosity, do MADlib developers regularly use code coverage
>>>>>> utilities to measure the coverage quality of their tests?
>>>>>>
>>>>>> -=e
>>>>>>
>>>>>> --
>>>>>> *Ed Espino*
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Ed Espino*
>>>>
>>>
>>
>>
>>
>> --
>> *Ed Espino*
>>
>
>
>
> --
> *Ed Espino*
Re: MADlib Code coverage
Posted by Ed Espino <es...@apache.org>.
Nandish,
For a very basic code coverage implementation using cmake, I have created:
* JIRA: https://issues.apache.org/jira/browse/MADLIB-1138
* PR: https://github.com/apache/incubator-madlib/pull/151
This is a basic code coverage implementation runs on my mac using the
following and a brew installed lcov module:
Configured with: --prefix=/Library/Developer/CommandLineTools/usr
--with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 8.1.0 (clang-802.0.42)
Target: x86_64-apple-darwin16.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
Regards,
-=e
On Fri, Jul 14, 2017 at 1:50 PM, Ed Espino <es...@apache.org> wrote:
> Sure NJ,
>
> I'll create both (Jira and PR).
>
> -=e
>
> On Fri, Jul 14, 2017 at 1:46 PM, Nandish Jayaram <nj...@pivotal.io>
> wrote:
>
>> Thank you Ed, this is great. We will create a JIRA to incorporate this
>> (including documenting the steps to use it). Or, you can also create a PR.
>> :)
>>
>> NJ
>>
>> On Fri, Jul 14, 2017 at 12:31 PM, Ed Espino <es...@apache.org> wrote:
>>
>> > NJ,
>> >
>> > I spent about an hour getting it setup on my mac. This supported the
>> > MLP PR review. I threw some info together below.
>> >
>> > Hope it helps,
>> > -=e
>> >
>> > ------------------
>> >
>> > Add lcov through brew (brings in lcov and genhtml utilities):
>> > brew install lcov
>> >
>> > "Total HACK" to add code coverage support into the build for C/C++
>> > code. This is quick and dirty so I could get to the data I needed for
>> > the PR review.
>> >
>> > diff --git a/CMakeLists.txt b/CMakeLists.txt
>> > index b2172ef3..db80986c 100644
>> > --- a/CMakeLists.txt
>> > +++ b/CMakeLists.txt
>> > @@ -104,7 +104,7 @@ if(CMAKE_COMPILER_IS_GNUCXX)
>> > set(CMAKE_INCLUDE_SYSTEM_FLAG_CXX "-isystem ")
>> > endif(APPLE)
>> > elseif(CMAKE_C_COMPILER_ID MATCHES "Clang")
>> > - set(CMAKE_CXX_FLAGS "-stdlib=libstdc++")
>> > + set(CMAKE_CXX_FLAGS "-stdlib=libstdc++ -fprofile-arcs
>> > -ftest-coverage")
>> > endif(CMAKE_COMPILER_IS_GNUCXX)
>> >
>> > # force a `m4_' prefix to all builtins
>> > @@ -114,6 +114,8 @@ else()
>> > set(M4_ARGUMENTS "--prefix-builtins")
>> > endif()
>> >
>> > +set(CMAKE_C_FLAGS "-fprofile-arcs -ftest-coverage")
>> > +
>> > # Read and parse Version.yml file
>> > file(READ "${MADLIB_VERSION_YML}" _MADLIB_VERSION_CONTENTS)
>> > string(REGEX REPLACE "^.*version:[ \t]*([^\n]*)\n.*" "\\1"
>> > MADLIB_VERSION_STRING "${_MADLIB_VERSION_CONTENTS}")
>> >
>> > Once MADlib is built and installed, run your tests:
>> >
>> > ## Build and install MADlib
>> > mkdir build
>> > cd build
>> > cmake ..
>> > make -j8 install
>> > /usr/local/madlib/bin/madpack -s madlib -p postgres install
>> >
>> > ## At this point you can run your tests (I'm focusing on MLP)
>> >
>> > /usr/local/madlib/bin/madpack -s madlib -p postgres install-check -t
>> > convex/mlp
>> >
>> > ## Time to capture the results and generate html report.
>> >
>> > lcov --capture --directory . --output-file coverage.info
>> > genhtml coverage.info --output-directory gcov
>> >
>> > # You will notice there is some uninteresting system coverage info.
>> > # The following will filter them out.
>> >
>> > lcov --remove coverage.info '/usr/include/*' '/usr/local/include/*'
>> > '/usr/local/postgres/*' '*build/third_party/*' -o
>> coverage_filtered.info
>> > rm -rf gcov
>> > genhtml coverage_filtered.info --output-directory gcov
>> >
>> > It is possible to zero out the counters with the following.
>> > lcov --zerocounters --directory .
>> >
>> >
>> > On Fri, Jul 14, 2017 at 12:23 PM, Nandish Jayaram <nj...@pivotal.io>
>> > wrote:
>> >
>> > > Hi Ed,
>> > >
>> > > We haven't set that up for MADlib yet, but we will be looking into it
>> > soon.
>> > > Any ideas?
>> > >
>> > > NJ
>> > >
>> > > On Fri, Jul 14, 2017 at 10:00 AM, Ed Espino <es...@apache.org>
>> wrote:
>> > >
>> > > > Out of curiosity, do MADlib developers regularly use code coverage
>> > > > utilities to measure the coverage quality of their tests?
>> > > >
>> > > > -=e
>> > > >
>> > > > --
>> > > > *Ed Espino*
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > *Ed Espino*
>> >
>>
>
>
>
> --
> *Ed Espino*
>
--
*Ed Espino*
Re: MADlib Code coverage
Posted by Ed Espino <es...@apache.org>.
Sure NJ,
I'll create both (Jira and PR).
-=e
On Fri, Jul 14, 2017 at 1:46 PM, Nandish Jayaram <nj...@pivotal.io>
wrote:
> Thank you Ed, this is great. We will create a JIRA to incorporate this
> (including documenting the steps to use it). Or, you can also create a PR.
> :)
>
> NJ
>
> On Fri, Jul 14, 2017 at 12:31 PM, Ed Espino <es...@apache.org> wrote:
>
> > NJ,
> >
> > I spent about an hour getting it setup on my mac. This supported the
> > MLP PR review. I threw some info together below.
> >
> > Hope it helps,
> > -=e
> >
> > ------------------
> >
> > Add lcov through brew (brings in lcov and genhtml utilities):
> > brew install lcov
> >
> > "Total HACK" to add code coverage support into the build for C/C++
> > code. This is quick and dirty so I could get to the data I needed for
> > the PR review.
> >
> > diff --git a/CMakeLists.txt b/CMakeLists.txt
> > index b2172ef3..db80986c 100644
> > --- a/CMakeLists.txt
> > +++ b/CMakeLists.txt
> > @@ -104,7 +104,7 @@ if(CMAKE_COMPILER_IS_GNUCXX)
> > set(CMAKE_INCLUDE_SYSTEM_FLAG_CXX "-isystem ")
> > endif(APPLE)
> > elseif(CMAKE_C_COMPILER_ID MATCHES "Clang")
> > - set(CMAKE_CXX_FLAGS "-stdlib=libstdc++")
> > + set(CMAKE_CXX_FLAGS "-stdlib=libstdc++ -fprofile-arcs
> > -ftest-coverage")
> > endif(CMAKE_COMPILER_IS_GNUCXX)
> >
> > # force a `m4_' prefix to all builtins
> > @@ -114,6 +114,8 @@ else()
> > set(M4_ARGUMENTS "--prefix-builtins")
> > endif()
> >
> > +set(CMAKE_C_FLAGS "-fprofile-arcs -ftest-coverage")
> > +
> > # Read and parse Version.yml file
> > file(READ "${MADLIB_VERSION_YML}" _MADLIB_VERSION_CONTENTS)
> > string(REGEX REPLACE "^.*version:[ \t]*([^\n]*)\n.*" "\\1"
> > MADLIB_VERSION_STRING "${_MADLIB_VERSION_CONTENTS}")
> >
> > Once MADlib is built and installed, run your tests:
> >
> > ## Build and install MADlib
> > mkdir build
> > cd build
> > cmake ..
> > make -j8 install
> > /usr/local/madlib/bin/madpack -s madlib -p postgres install
> >
> > ## At this point you can run your tests (I'm focusing on MLP)
> >
> > /usr/local/madlib/bin/madpack -s madlib -p postgres install-check -t
> > convex/mlp
> >
> > ## Time to capture the results and generate html report.
> >
> > lcov --capture --directory . --output-file coverage.info
> > genhtml coverage.info --output-directory gcov
> >
> > # You will notice there is some uninteresting system coverage info.
> > # The following will filter them out.
> >
> > lcov --remove coverage.info '/usr/include/*' '/usr/local/include/*'
> > '/usr/local/postgres/*' '*build/third_party/*' -o coverage_filtered.info
> > rm -rf gcov
> > genhtml coverage_filtered.info --output-directory gcov
> >
> > It is possible to zero out the counters with the following.
> > lcov --zerocounters --directory .
> >
> >
> > On Fri, Jul 14, 2017 at 12:23 PM, Nandish Jayaram <nj...@pivotal.io>
> > wrote:
> >
> > > Hi Ed,
> > >
> > > We haven't set that up for MADlib yet, but we will be looking into it
> > soon.
> > > Any ideas?
> > >
> > > NJ
> > >
> > > On Fri, Jul 14, 2017 at 10:00 AM, Ed Espino <es...@apache.org> wrote:
> > >
> > > > Out of curiosity, do MADlib developers regularly use code coverage
> > > > utilities to measure the coverage quality of their tests?
> > > >
> > > > -=e
> > > >
> > > > --
> > > > *Ed Espino*
> > > >
> > >
> >
> >
> >
> > --
> > *Ed Espino*
> >
>
--
*Ed Espino*
Re: MADlib Code coverage
Posted by Nandish Jayaram <nj...@pivotal.io>.
Thank you Ed, this is great. We will create a JIRA to incorporate this
(including documenting the steps to use it). Or, you can also create a PR.
:)
NJ
On Fri, Jul 14, 2017 at 12:31 PM, Ed Espino <es...@apache.org> wrote:
> NJ,
>
> I spent about an hour getting it setup on my mac. This supported the
> MLP PR review. I threw some info together below.
>
> Hope it helps,
> -=e
>
> ------------------
>
> Add lcov through brew (brings in lcov and genhtml utilities):
> brew install lcov
>
> "Total HACK" to add code coverage support into the build for C/C++
> code. This is quick and dirty so I could get to the data I needed for
> the PR review.
>
> diff --git a/CMakeLists.txt b/CMakeLists.txt
> index b2172ef3..db80986c 100644
> --- a/CMakeLists.txt
> +++ b/CMakeLists.txt
> @@ -104,7 +104,7 @@ if(CMAKE_COMPILER_IS_GNUCXX)
> set(CMAKE_INCLUDE_SYSTEM_FLAG_CXX "-isystem ")
> endif(APPLE)
> elseif(CMAKE_C_COMPILER_ID MATCHES "Clang")
> - set(CMAKE_CXX_FLAGS "-stdlib=libstdc++")
> + set(CMAKE_CXX_FLAGS "-stdlib=libstdc++ -fprofile-arcs
> -ftest-coverage")
> endif(CMAKE_COMPILER_IS_GNUCXX)
>
> # force a `m4_' prefix to all builtins
> @@ -114,6 +114,8 @@ else()
> set(M4_ARGUMENTS "--prefix-builtins")
> endif()
>
> +set(CMAKE_C_FLAGS "-fprofile-arcs -ftest-coverage")
> +
> # Read and parse Version.yml file
> file(READ "${MADLIB_VERSION_YML}" _MADLIB_VERSION_CONTENTS)
> string(REGEX REPLACE "^.*version:[ \t]*([^\n]*)\n.*" "\\1"
> MADLIB_VERSION_STRING "${_MADLIB_VERSION_CONTENTS}")
>
> Once MADlib is built and installed, run your tests:
>
> ## Build and install MADlib
> mkdir build
> cd build
> cmake ..
> make -j8 install
> /usr/local/madlib/bin/madpack -s madlib -p postgres install
>
> ## At this point you can run your tests (I'm focusing on MLP)
>
> /usr/local/madlib/bin/madpack -s madlib -p postgres install-check -t
> convex/mlp
>
> ## Time to capture the results and generate html report.
>
> lcov --capture --directory . --output-file coverage.info
> genhtml coverage.info --output-directory gcov
>
> # You will notice there is some uninteresting system coverage info.
> # The following will filter them out.
>
> lcov --remove coverage.info '/usr/include/*' '/usr/local/include/*'
> '/usr/local/postgres/*' '*build/third_party/*' -o coverage_filtered.info
> rm -rf gcov
> genhtml coverage_filtered.info --output-directory gcov
>
> It is possible to zero out the counters with the following.
> lcov --zerocounters --directory .
>
>
> On Fri, Jul 14, 2017 at 12:23 PM, Nandish Jayaram <nj...@pivotal.io>
> wrote:
>
> > Hi Ed,
> >
> > We haven't set that up for MADlib yet, but we will be looking into it
> soon.
> > Any ideas?
> >
> > NJ
> >
> > On Fri, Jul 14, 2017 at 10:00 AM, Ed Espino <es...@apache.org> wrote:
> >
> > > Out of curiosity, do MADlib developers regularly use code coverage
> > > utilities to measure the coverage quality of their tests?
> > >
> > > -=e
> > >
> > > --
> > > *Ed Espino*
> > >
> >
>
>
>
> --
> *Ed Espino*
>
Re: MADlib Code coverage
Posted by Ed Espino <es...@apache.org>.
NJ,
I spent about an hour getting it setup on my mac. This supported the
MLP PR review. I threw some info together below.
Hope it helps,
-=e
------------------
Add lcov through brew (brings in lcov and genhtml utilities):
brew install lcov
"Total HACK" to add code coverage support into the build for C/C++
code. This is quick and dirty so I could get to the data I needed for
the PR review.
diff --git a/CMakeLists.txt b/CMakeLists.txt
index b2172ef3..db80986c 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -104,7 +104,7 @@ if(CMAKE_COMPILER_IS_GNUCXX)
set(CMAKE_INCLUDE_SYSTEM_FLAG_CXX "-isystem ")
endif(APPLE)
elseif(CMAKE_C_COMPILER_ID MATCHES "Clang")
- set(CMAKE_CXX_FLAGS "-stdlib=libstdc++")
+ set(CMAKE_CXX_FLAGS "-stdlib=libstdc++ -fprofile-arcs
-ftest-coverage")
endif(CMAKE_COMPILER_IS_GNUCXX)
# force a `m4_' prefix to all builtins
@@ -114,6 +114,8 @@ else()
set(M4_ARGUMENTS "--prefix-builtins")
endif()
+set(CMAKE_C_FLAGS "-fprofile-arcs -ftest-coverage")
+
# Read and parse Version.yml file
file(READ "${MADLIB_VERSION_YML}" _MADLIB_VERSION_CONTENTS)
string(REGEX REPLACE "^.*version:[ \t]*([^\n]*)\n.*" "\\1"
MADLIB_VERSION_STRING "${_MADLIB_VERSION_CONTENTS}")
Once MADlib is built and installed, run your tests:
## Build and install MADlib
mkdir build
cd build
cmake ..
make -j8 install
/usr/local/madlib/bin/madpack -s madlib -p postgres install
## At this point you can run your tests (I'm focusing on MLP)
/usr/local/madlib/bin/madpack -s madlib -p postgres install-check -t
convex/mlp
## Time to capture the results and generate html report.
lcov --capture --directory . --output-file coverage.info
genhtml coverage.info --output-directory gcov
# You will notice there is some uninteresting system coverage info.
# The following will filter them out.
lcov --remove coverage.info '/usr/include/*' '/usr/local/include/*'
'/usr/local/postgres/*' '*build/third_party/*' -o coverage_filtered.info
rm -rf gcov
genhtml coverage_filtered.info --output-directory gcov
It is possible to zero out the counters with the following.
lcov --zerocounters --directory .
On Fri, Jul 14, 2017 at 12:23 PM, Nandish Jayaram <nj...@pivotal.io>
wrote:
> Hi Ed,
>
> We haven't set that up for MADlib yet, but we will be looking into it soon.
> Any ideas?
>
> NJ
>
> On Fri, Jul 14, 2017 at 10:00 AM, Ed Espino <es...@apache.org> wrote:
>
> > Out of curiosity, do MADlib developers regularly use code coverage
> > utilities to measure the coverage quality of their tests?
> >
> > -=e
> >
> > --
> > *Ed Espino*
> >
>
--
*Ed Espino*
Re: MADlib Code coverage
Posted by Nandish Jayaram <nj...@pivotal.io>.
Hi Ed,
We haven't set that up for MADlib yet, but we will be looking into it soon.
Any ideas?
NJ
On Fri, Jul 14, 2017 at 10:00 AM, Ed Espino <es...@apache.org> wrote:
> Out of curiosity, do MADlib developers regularly use code coverage
> utilities to measure the coverage quality of their tests?
>
> -=e
>
> --
> *Ed Espino*
>