You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "Sai Zhang (JIRA)" <ji...@apache.org> on 2011/04/03 10:04:05 UTC
[jira] [Created] (COLLECTIONS-374) Bug in class#BeanMap and
TransformedBuffer with reproducible JUnit test
Bug in class#BeanMap and TransformedBuffer with reproducible JUnit test
-----------------------------------------------------------------------
Key: COLLECTIONS-374
URL: https://issues.apache.org/jira/browse/COLLECTIONS-374
Project: Commons Collections
Issue Type: Bug
Affects Versions: 3.2
Environment: jdk 1.6.0
Reporter: Sai Zhang
Attachments: ApacheCommons_Documented_Test.java
Hi all:
(as in the previous post)
I am writing an automated bug finding tool, and using
Apache Commons Collections as an experimental subject
for evaluation.
The tool creates executable JUnit tests as well as
explanatory code comments. I attached one bug-revealing
test as follows. Could you please kindly check it, to
see if it is a real bug or not?
Also, it would be tremendous helpful if you could give
some feedback and suggestion on the generated code comments?
Is the automatically-inferred comment useful in understanding
the generated test? is the comment helpful in bug fixing?
Your suggestion will help us improve the tool.
Please see attachment for the failed test.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COLLECTIONS-374) Bug in class#BeanMap and
TransformedBuffer with reproducible JUnit test
Posted by "Sebb (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COLLECTIONS-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015191#comment-13015191 ]
Sebb commented on COLLECTIONS-374:
----------------------------------
The assertion itself is easy to understand, it's just that all the redundant assignments make it hard to read (and may generate warnings from some IDEs).
> Bug in class#BeanMap and TransformedBuffer with reproducible JUnit test
> -----------------------------------------------------------------------
>
> Key: COLLECTIONS-374
> URL: https://issues.apache.org/jira/browse/COLLECTIONS-374
> Project: Commons Collections
> Issue Type: Bug
> Affects Versions: 3.2
> Environment: jdk 1.6.0
> Reporter: Sai Zhang
> Attachments: ApacheCommons_Documented_Test.java
>
>
> Hi all:
> (as in the previous post)
> I am writing an automated bug finding tool, and using
> Apache Commons Collections as an experimental subject
> for evaluation.
> The tool creates executable JUnit tests as well as
> explanatory code comments. I attached one bug-revealing
> test as follows. Could you please kindly check it, to
> see if it is a real bug or not?
> Also, it would be tremendous helpful if you could give
> some feedback and suggestion on the generated code comments?
> From the perspective of developers who are relatively familiar
> with the code,
> is the automatically-inferred comment useful in understanding
> the generated test? is the comment helpful in bug fixing?
> Your suggestion will help us improve the tool.
> Please see attachment for the failed test.
> The comment appears in the form of:
> //Tests pass if .... (it gives some small change to the test which can make the failed test pass)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COLLECTIONS-374) Bug in class#BeanMap and
TransformedBuffer with reproducible JUnit test
Posted by "Sebb (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COLLECTIONS-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015127#comment-13015127 ]
Sebb commented on COLLECTIONS-374:
----------------------------------
I should have added - test1() does seem to be valid.
Not sure whether test0() is useful, given that the class is deprecated, but it does seem to expose a failure, as the hashCode is not stable.
> Bug in class#BeanMap and TransformedBuffer with reproducible JUnit test
> -----------------------------------------------------------------------
>
> Key: COLLECTIONS-374
> URL: https://issues.apache.org/jira/browse/COLLECTIONS-374
> Project: Commons Collections
> Issue Type: Bug
> Affects Versions: 3.2
> Environment: jdk 1.6.0
> Reporter: Sai Zhang
> Attachments: ApacheCommons_Documented_Test.java
>
>
> Hi all:
> (as in the previous post)
> I am writing an automated bug finding tool, and using
> Apache Commons Collections as an experimental subject
> for evaluation.
> The tool creates executable JUnit tests as well as
> explanatory code comments. I attached one bug-revealing
> test as follows. Could you please kindly check it, to
> see if it is a real bug or not?
> Also, it would be tremendous helpful if you could give
> some feedback and suggestion on the generated code comments?
> From the perspective of developers who are relatively familiar
> with the code,
> is the automatically-inferred comment useful in understanding
> the generated test? is the comment helpful in bug fixing?
> Your suggestion will help us improve the tool.
> Please see attachment for the failed test.
> The comment appears in the form of:
> //Tests pass if .... (it gives some small change to the test which can make the failed test pass)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COLLECTIONS-374) Bug in class#BeanMap and
TransformedBuffer with reproducible JUnit test
Posted by "Sai Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COLLECTIONS-374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sai Zhang updated COLLECTIONS-374:
----------------------------------
Attachment: ApacheCommons_Documented_Test.java
> Bug in class#BeanMap and TransformedBuffer with reproducible JUnit test
> -----------------------------------------------------------------------
>
> Key: COLLECTIONS-374
> URL: https://issues.apache.org/jira/browse/COLLECTIONS-374
> Project: Commons Collections
> Issue Type: Bug
> Affects Versions: 3.2
> Environment: jdk 1.6.0
> Reporter: Sai Zhang
> Attachments: ApacheCommons_Documented_Test.java
>
>
> Hi all:
> (as in the previous post)
> I am writing an automated bug finding tool, and using
> Apache Commons Collections as an experimental subject
> for evaluation.
> The tool creates executable JUnit tests as well as
> explanatory code comments. I attached one bug-revealing
> test as follows. Could you please kindly check it, to
> see if it is a real bug or not?
> Also, it would be tremendous helpful if you could give
> some feedback and suggestion on the generated code comments?
> Is the automatically-inferred comment useful in understanding
> the generated test? is the comment helpful in bug fixing?
> Your suggestion will help us improve the tool.
> Please see attachment for the failed test.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COLLECTIONS-374) Bug in class#BeanMap and
TransformedBuffer with reproducible JUnit test
Posted by "Sai Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COLLECTIONS-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015165#comment-13015165 ]
Sai Zhang commented on COLLECTIONS-374:
---------------------------------------
Thank you very much, Sebb, for all your good suggestion!
We really appreciate your response.
Thanks. We should implement this feature.
Thanks. We should improve the readability of class name
Actually, all you have mentioned above reflect the fact that:
automatically-generated test, though
can reveal previously-unknown bugs, is hard to interpret. From the viewpoint
of developing new fully-automatic testing techniques, that is an inherent
problem, because to reveal bugs: the test created need to be
behaviorally-diverse (e.g., covering as many program states as possible).
Therefore, in my tool, we use several heuristic and randomized algorithms to
achieve this (since doing exhaustive program state search is infeasible,
given the huge space of possible method invocations).
The comments the tool generates aim to alleviate (we can not say it solves)
the above problem (poor readability). As you may find, the generated test is
long, and often has many unused variables. Even developers who are
already familiar with the code can not easily have ideas on which test code
part should they inspect. The comments provide an alternative way to
"correct" a failed test, which we hope to given additional debugging clues.
We add this "comment " feature based on our own (limited) experience: when
given a long/hard-to-read failed test, a common practice for programmers to
start debugging is try to make some minimal edit, making the failed test
pass. Then, observe the difference between a failed and passing execution.
We agree that the tool itself is still far from perfect (due to the
randomized algorithm it uses). Compared with the long test without comments,
do you think the test with comments can somehow give certain debugging
clues, and help to guide programmers to inspect the right place more
efficiently? (we know the automatically-generated test is still much worse
than human written one)
Thanks a lot.
-Sai
> Bug in class#BeanMap and TransformedBuffer with reproducible JUnit test
> -----------------------------------------------------------------------
>
> Key: COLLECTIONS-374
> URL: https://issues.apache.org/jira/browse/COLLECTIONS-374
> Project: Commons Collections
> Issue Type: Bug
> Affects Versions: 3.2
> Environment: jdk 1.6.0
> Reporter: Sai Zhang
> Attachments: ApacheCommons_Documented_Test.java
>
>
> Hi all:
> (as in the previous post)
> I am writing an automated bug finding tool, and using
> Apache Commons Collections as an experimental subject
> for evaluation.
> The tool creates executable JUnit tests as well as
> explanatory code comments. I attached one bug-revealing
> test as follows. Could you please kindly check it, to
> see if it is a real bug or not?
> Also, it would be tremendous helpful if you could give
> some feedback and suggestion on the generated code comments?
> From the perspective of developers who are relatively familiar
> with the code,
> is the automatically-inferred comment useful in understanding
> the generated test? is the comment helpful in bug fixing?
> Your suggestion will help us improve the tool.
> Please see attachment for the failed test.
> The comment appears in the form of:
> //Tests pass if .... (it gives some small change to the test which can make the failed test pass)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COLLECTIONS-374) Bug in class#BeanMap and
TransformedBuffer with reproducible JUnit test
Posted by "Sai Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COLLECTIONS-374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sai Zhang updated COLLECTIONS-374:
----------------------------------
Description:
Hi all:
(as in the previous post)
I am writing an automated bug finding tool, and using
Apache Commons Collections as an experimental subject
for evaluation.
The tool creates executable JUnit tests as well as
explanatory code comments. I attached one bug-revealing
test as follows. Could you please kindly check it, to
see if it is a real bug or not?
Also, it would be tremendous helpful if you could give
some feedback and suggestion on the generated code comments?
>From the perspective of developers who are relatively familiar
with the code,
is the automatically-inferred comment useful in understanding
the generated test? is the comment helpful in bug fixing?
Your suggestion will help us improve the tool.
Please see attachment for the failed test.
The comment appears in the form of:
//Tests pass if .... (it gives some small change to the test which can make the failed test pass)
was:
Hi all:
(as in the previous post)
I am writing an automated bug finding tool, and using
Apache Commons Collections as an experimental subject
for evaluation.
The tool creates executable JUnit tests as well as
explanatory code comments. I attached one bug-revealing
test as follows. Could you please kindly check it, to
see if it is a real bug or not?
Also, it would be tremendous helpful if you could give
some feedback and suggestion on the generated code comments?
Is the automatically-inferred comment useful in understanding
the generated test? is the comment helpful in bug fixing?
Your suggestion will help us improve the tool.
Please see attachment for the failed test.
> Bug in class#BeanMap and TransformedBuffer with reproducible JUnit test
> -----------------------------------------------------------------------
>
> Key: COLLECTIONS-374
> URL: https://issues.apache.org/jira/browse/COLLECTIONS-374
> Project: Commons Collections
> Issue Type: Bug
> Affects Versions: 3.2
> Environment: jdk 1.6.0
> Reporter: Sai Zhang
> Attachments: ApacheCommons_Documented_Test.java
>
>
> Hi all:
> (as in the previous post)
> I am writing an automated bug finding tool, and using
> Apache Commons Collections as an experimental subject
> for evaluation.
> The tool creates executable JUnit tests as well as
> explanatory code comments. I attached one bug-revealing
> test as follows. Could you please kindly check it, to
> see if it is a real bug or not?
> Also, it would be tremendous helpful if you could give
> some feedback and suggestion on the generated code comments?
> From the perspective of developers who are relatively familiar
> with the code,
> is the automatically-inferred comment useful in understanding
> the generated test? is the comment helpful in bug fixing?
> Your suggestion will help us improve the tool.
> Please see attachment for the failed test.
> The comment appears in the form of:
> //Tests pass if .... (it gives some small change to the test which can make the failed test pass)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COLLECTIONS-374) Bug in class#BeanMap and
TransformedBuffer with reproducible JUnit test
Posted by "Sebb (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COLLECTIONS-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015125#comment-13015125 ]
Sebb commented on COLLECTIONS-374:
----------------------------------
The tool sounds interesting, but at present it is difficult to check what is happening. The comments on the asserts are OK, but otherwise the tests are hard to understand.
Every class name is given in full, even java.lang.Integer. The testing tool should generate the necessary imports.
There should be a package clause with the same package as the class under test, and the test class name should relate to the class under test, e.g. BeanMapAutoTest. Each test class should only test one target class.
The throws clauses should be "throws Exception" rather than Throwable.
In the test0() method, there appear to be lots of unused variables, and the test target (BeanMap) has been deprecated (and in fact has since been removed).
In the test1() method, again there are a lot of unused variables. It also looks like the tool failed to add a test part way through, as there is a comment, but no assertion. This might explain the unused variables. Ideally every assertion should be in a different method.
> Bug in class#BeanMap and TransformedBuffer with reproducible JUnit test
> -----------------------------------------------------------------------
>
> Key: COLLECTIONS-374
> URL: https://issues.apache.org/jira/browse/COLLECTIONS-374
> Project: Commons Collections
> Issue Type: Bug
> Affects Versions: 3.2
> Environment: jdk 1.6.0
> Reporter: Sai Zhang
> Attachments: ApacheCommons_Documented_Test.java
>
>
> Hi all:
> (as in the previous post)
> I am writing an automated bug finding tool, and using
> Apache Commons Collections as an experimental subject
> for evaluation.
> The tool creates executable JUnit tests as well as
> explanatory code comments. I attached one bug-revealing
> test as follows. Could you please kindly check it, to
> see if it is a real bug or not?
> Also, it would be tremendous helpful if you could give
> some feedback and suggestion on the generated code comments?
> From the perspective of developers who are relatively familiar
> with the code,
> is the automatically-inferred comment useful in understanding
> the generated test? is the comment helpful in bug fixing?
> Your suggestion will help us improve the tool.
> Please see attachment for the failed test.
> The comment appears in the form of:
> //Tests pass if .... (it gives some small change to the test which can make the failed test pass)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COLLECTIONS-374) Bug in class#BeanMap and
TransformedBuffer with reproducible JUnit test
Posted by "Sai Zhang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COLLECTIONS-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015194#comment-13015194 ]
Sai Zhang commented on COLLECTIONS-374:
---------------------------------------
Thanks Sebb. What about the comments in the middle of the long test. Do you
think such kind of comments useful for debugging?
-Sai
--
> Bug in class#BeanMap and TransformedBuffer with reproducible JUnit test
> -----------------------------------------------------------------------
>
> Key: COLLECTIONS-374
> URL: https://issues.apache.org/jira/browse/COLLECTIONS-374
> Project: Commons Collections
> Issue Type: Bug
> Affects Versions: 3.2
> Environment: jdk 1.6.0
> Reporter: Sai Zhang
> Attachments: ApacheCommons_Documented_Test.java
>
>
> Hi all:
> (as in the previous post)
> I am writing an automated bug finding tool, and using
> Apache Commons Collections as an experimental subject
> for evaluation.
> The tool creates executable JUnit tests as well as
> explanatory code comments. I attached one bug-revealing
> test as follows. Could you please kindly check it, to
> see if it is a real bug or not?
> Also, it would be tremendous helpful if you could give
> some feedback and suggestion on the generated code comments?
> From the perspective of developers who are relatively familiar
> with the code,
> is the automatically-inferred comment useful in understanding
> the generated test? is the comment helpful in bug fixing?
> Your suggestion will help us improve the tool.
> Please see attachment for the failed test.
> The comment appears in the form of:
> //Tests pass if .... (it gives some small change to the test which can make the failed test pass)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira