Master Thesis on False Positives in Test Failures

[Kevin van den Bekerom <k....@sig.eu>]

Dear Developers of the Apache Accumulo project,



My name is Kevin van den Bekerom and I am currently doing my Master's
research on the topic of false alarms in test code. I would like to ask the
input of the Accumulo development team categorizing test code bugs.



My research is based on a recent paper by Arash et al. (
http://salt.ece.ubc.ca/publications/docs/icsme15.pdf). They conducted an
empirical study, categorizing "test code bugs" in Apache software projects,
e.g. semantic, flaky, environmental, etc. A "test code bug" is a failing
test, where the System Under Test is correct, but the test code is
incorrect. To identify test code bugs they looked at issues in JIRA, and
checked if the fixing commit was only in the test code. Only fixed issues
were counted and categorised.



My goal is to replicate their results using a different approach, i.e. ask
developers that were involved in the issue and/or fix how they would
categorize it.  For the Accumulo project they counted 187 test code bugs.
Insight into false positives can therefore be very relevant for your
project. Note that they only sampled a number of identified test code bugs
for individual inspection (30 for the Accumulo project).


I would like to ask the Accumulo team’s participation in categorizing the
various test code bugs. I will provide a list of JIRA IDs which are
identified as test code bugs and an initial list of categories to aid in
the categorization process. In my belief, the developers that worked on the
issue are the one's that are most capable of categorizing the issue. Please
let me know if this project looks interesting to you and you are willing to
help me out.



As a next step I will look for common patterns in identified test code bugs
and my aim is to extent static source code analysis techniques to be also
suited to find test code bugs. I am of course very happy to share my
findings with the team.



Hope to hear from you!



With kind regards,

-- 
*Kevin van den Bekerom* | Intern

+31 6 21 33 93 85 | kvandenbekerom@sig.eu
Software Improvement Group | www.sig.eu

Re: Master Thesis on False Positives in Test Failures

[Josh Elser <jo...@gmail.com>]

Hi Kevin,

Many of those test bugs and fixes were probably my doing.

Most of them were just flakiness in general, but, if you can provide an 
explicit list, I can try to confirm whether or not that was exactly the 
case.

- Josh

Kevin van den Bekerom wrote:
> Dear Developers of the Apache Accumulo project,
>
>
>
> My name is Kevin van den Bekerom and I am currently doing my Master's
> research on the topic of false alarms in test code. I would like to ask the
> input of the Accumulo development team categorizing test code bugs.
>
>
>
> My research is based on a recent paper by Arash et al. (
> http://salt.ece.ubc.ca/publications/docs/icsme15.pdf). They conducted an
> empirical study, categorizing "test code bugs" in Apache software projects,
> e.g. semantic, flaky, environmental, etc. A "test code bug" is a failing
> test, where the System Under Test is correct, but the test code is
> incorrect. To identify test code bugs they looked at issues in JIRA, and
> checked if the fixing commit was only in the test code. Only fixed issues
> were counted and categorised.
>
>
>
> My goal is to replicate their results using a different approach, i.e. ask
> developers that were involved in the issue and/or fix how they would
> categorize it.  For the Accumulo project they counted 187 test code bugs.
> Insight into false positives can therefore be very relevant for your
> project. Note that they only sampled a number of identified test code bugs
> for individual inspection (30 for the Accumulo project).
>
>
> I would like to ask the Accumulo team’s participation in categorizing the
> various test code bugs. I will provide a list of JIRA IDs which are
> identified as test code bugs and an initial list of categories to aid in
> the categorization process. In my belief, the developers that worked on the
> issue are the one's that are most capable of categorizing the issue. Please
> let me know if this project looks interesting to you and you are willing to
> help me out.
>
>
>
> As a next step I will look for common patterns in identified test code bugs
> and my aim is to extent static source code analysis techniques to be also
> suited to find test code bugs. I am of course very happy to share my
> findings with the team.
>
>
>
> Hope to hear from you!
>
>
>
> With kind regards,
>