You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@harmony.apache.org by "Alexander Kleymenov (JIRA)" <ji...@apache.org> on 2006/12/08 12:45:21 UTC
[jira] Created: (HARMONY-2540) [drlvm][testing] Regression Testing
Infrastructure implementation.
[drlvm][testing] Regression Testing Infrastructure implementation.
------------------------------------------------------------------
Key: HARMONY-2540
URL: http://issues.apache.org/jira/browse/HARMONY-2540
Project: Harmony
Issue Type: Improvement
Components: DRLVM
Reporter: Alexander Kleymenov
At present the using of DRLVM regression testing is complicated by the following reasons:
- It does not support automated test integration: integration of new regression test requires changes in build system.
- It does not contain the document describing the directions for developers of new regression tests.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HARMONY-2540) [drlvm][testing] Regression
Testing Infrastructure implementation.
Posted by "Alexey Varlamov (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HARMONY-2540?page=comments#action_12460976 ]
Alexey Varlamov commented on HARMONY-2540:
------------------------------------------
Alexander, your latest patch basically undoes my modifications for uncoupling reg.test targets with super-duper meta-component-designed drlvm build system, and removes regression/H0000 test. I did both mods to your original patch deliberately, targeting greater simplicity and painless transition to better build infrastructure in future. The H0000 test is a nice grindstone for the regression framework so I prefer to keep it fot now.
The only essence improvement I see is the following bit:
Index: build/make/targets/reg.test.run.xml
===================================================================
--- build/make/targets/reg.test.run.xml (revision 490268)
+++ build/make/targets/reg.test.run.xml (working copy)
@@ -146,9 +146,10 @@
<junit jvm="${jvm.under.test}"
fork="yes"
haltonfailure="false"
- haltonerror="false"
+ haltonerror="${reg.test.halt.on.error}"
printsummary="withOutAndErr"
dir="${basedir}"
+ errorproperty="reg.test.errors"
failureproperty="reg.test.errors"
timeout="${reg.test.timeout}">
<jvmarg line="@{vmarg}"/>
@@ -160,9 +161,19 @@
<formatter type="brief"/>
<formatter type="xml"/>
</junit>
+ <if>
+ <isset property="reg.test.custom.launched"/>
+ <then>
+ <fail message="${reg.test.failed.mark}">
+ <condition>
+ <istrue value="${reg.test.errors}"/>
+ </condition>
+ </fail>
+ </then>
+ </if>
</sequential>
</macrodef>
BTW, the last one is a good sample to improve ant-scripting style. Compare to the following alternative:
@@ -160,9 +161,10 @@
<formatter type="brief"/>
<formatter type="xml"/>
</junit>
+ <fail message="${reg.test.failed.mark}" if="${reg.test.errors}"/>
</sequential>
</macrodef>
> [drlvm][testing] Regression Testing Infrastructure implementation.
> ------------------------------------------------------------------
>
> Key: HARMONY-2540
> URL: http://issues.apache.org/jira/browse/HARMONY-2540
> Project: Harmony
> Issue Type: Improvement
> Components: DRLVM
> Reporter: Alexander Kleymenov
> Assigned To: Alexey Varlamov
> Attachments: patch-reg-test-fwk.txt, patch-regression-testing.txt
>
>
> At present the using of DRLVM regression testing is complicated by the following reasons:
> - It does not support automated test integration: integration of new regression test requires changes in build system.
> - It does not contain the document describing the directions for developers of new regression tests.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HARMONY-2540) [drlvm][testing] Regression
Testing Infrastructure implementation.
Posted by "Alexey Varlamov (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HARMONY-2540?page=comments#action_12461161 ]
Alexey Varlamov commented on HARMONY-2540:
------------------------------------------
Gosh, I wish our talk went in different tone. My committership does not make me smarter than you or any other participant here, just imposes additional responsibility; I hope I did not indicate otherwise in any of my comments. OTOH you cannot force me blindly commit anything.
So we have to come in agreement and collaborate.
> [drlvm][testing] Regression Testing Infrastructure implementation.
> ------------------------------------------------------------------
>
> Key: HARMONY-2540
> URL: http://issues.apache.org/jira/browse/HARMONY-2540
> Project: Harmony
> Issue Type: Improvement
> Components: DRLVM
> Reporter: Alexander Kleymenov
> Assigned To: Alexey Varlamov
> Attachments: patch-reg-test-fwk.txt, patch-regression-testing.txt
>
>
> At present the using of DRLVM regression testing is complicated by the following reasons:
> - It does not support automated test integration: integration of new regression test requires changes in build system.
> - It does not contain the document describing the directions for developers of new regression tests.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (HARMONY-2540) [drlvm][testing] Regression Testing
Infrastructure implementation.
Posted by "Alexey Varlamov (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HARMONY-2540?page=all ]
Alexey Varlamov resolved HARMONY-2540.
--------------------------------------
Resolution: Fixed
Alexander, thanks for the improvements.
The patch did not apply cleanly, so I merged it manually, and simplified some bits.
The system still needs some love, but now we can make better progress with adding tests.
Minor notices for the future: please try to avoid so tight coupling of independent patches.
Applied at r489018, please verify.
> [drlvm][testing] Regression Testing Infrastructure implementation.
> ------------------------------------------------------------------
>
> Key: HARMONY-2540
> URL: http://issues.apache.org/jira/browse/HARMONY-2540
> Project: Harmony
> Issue Type: Improvement
> Components: DRLVM
> Reporter: Alexander Kleymenov
> Assigned To: Alexey Varlamov
> Attachments: patch-regression-testing.txt
>
>
> At present the using of DRLVM regression testing is complicated by the following reasons:
> - It does not support automated test integration: integration of new regression test requires changes in build system.
> - It does not contain the document describing the directions for developers of new regression tests.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HARMONY-2540) [drlvm][testing] Regression
Testing Infrastructure implementation.
Posted by "Alexey Varlamov (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HARMONY-2540?page=comments#action_12461162 ]
Alexey Varlamov commented on HARMONY-2540:
------------------------------------------
Extra spaces came from manual merging, sorry if this disturbs you. Should we beautify the files (say, by Eclipse autoformatter) before going on?
> [drlvm][testing] Regression Testing Infrastructure implementation.
> ------------------------------------------------------------------
>
> Key: HARMONY-2540
> URL: http://issues.apache.org/jira/browse/HARMONY-2540
> Project: Harmony
> Issue Type: Improvement
> Components: DRLVM
> Reporter: Alexander Kleymenov
> Assigned To: Alexey Varlamov
> Attachments: patch-reg-test-fwk.txt, patch-regression-testing.txt
>
>
> At present the using of DRLVM regression testing is complicated by the following reasons:
> - It does not support automated test integration: integration of new regression test requires changes in build system.
> - It does not contain the document describing the directions for developers of new regression tests.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HARMONY-2540) [drlvm][testing] Regression Testing
Infrastructure implementation.
Posted by "Alexander Kleymenov (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HARMONY-2540?page=all ]
Alexander Kleymenov updated HARMONY-2540:
-----------------------------------------
Attachment: patch-reg-test-fwk.txt
Alexey,
There are some problems with result of patch application. Please find attached fixes to eliminate them. If you have any questions regarding the fix, please, don't hesistate to ask.
Thanks,
Alexander
> [drlvm][testing] Regression Testing Infrastructure implementation.
> ------------------------------------------------------------------
>
> Key: HARMONY-2540
> URL: http://issues.apache.org/jira/browse/HARMONY-2540
> Project: Harmony
> Issue Type: Improvement
> Components: DRLVM
> Reporter: Alexander Kleymenov
> Assigned To: Alexey Varlamov
> Attachments: patch-reg-test-fwk.txt, patch-regression-testing.txt
>
>
> At present the using of DRLVM regression testing is complicated by the following reasons:
> - It does not support automated test integration: integration of new regression test requires changes in build system.
> - It does not contain the document describing the directions for developers of new regression tests.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HARMONY-2540) [drlvm][testing] Regression Testing
Infrastructure implementation.
Posted by "Alexander Kleymenov (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HARMONY-2540?page=all ]
Alexander Kleymenov updated HARMONY-2540:
-----------------------------------------
Attachment: patch-regression-testing.txt
Attached patch finally completes the works on regression testing infrastructure.
It implements automated new test integration and README file describing
the directions for developers of new regression tests.
With this patch regression testing infrastructure is ready to be used
for automated regression testing and for implementation of new
regression tests.
This patch should be applied on top of the patch supplied with HARMONY-2416 report.
> [drlvm][testing] Regression Testing Infrastructure implementation.
> ------------------------------------------------------------------
>
> Key: HARMONY-2540
> URL: http://issues.apache.org/jira/browse/HARMONY-2540
> Project: Harmony
> Issue Type: Improvement
> Components: DRLVM
> Reporter: Alexander Kleymenov
> Attachments: patch-regression-testing.txt
>
>
> At present the using of DRLVM regression testing is complicated by the following reasons:
> - It does not support automated test integration: integration of new regression test requires changes in build system.
> - It does not contain the document describing the directions for developers of new regression tests.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HARMONY-2540) [drlvm][testing] Regression
Testing Infrastructure implementation.
Posted by "Alexander Kleymenov (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HARMONY-2540?page=comments#action_12461163 ]
Alexander Kleymenov commented on HARMONY-2540:
----------------------------------------------
> Gosh, I wish our talk went in different tone.
Hm, I didn't notice.
> My committership does not
> make me smarter than you or any other participant here, just imposes
> additional responsibility; I hope I did not indicate otherwise in any of
> my comments. OTOH you cannot force me blindly commit anything.
I didn't ever think to force you. You took it by yourself.
Please, don't change the discussion track.
> So we have to come in agreement and collaborate.
Yes, I'm trying to do so. Let's do it together :)
> Extra spaces came from manual merging, sorry if this disturbs you.
OK. We should comply with code conventions.
> Should we beautify the files (say, by Eclipse autoformatter) before
> going on?
No, just apply my fixes.
If you want do some build beautifying it can be done by another JIRA.
Alexander
> [drlvm][testing] Regression Testing Infrastructure implementation.
> ------------------------------------------------------------------
>
> Key: HARMONY-2540
> URL: http://issues.apache.org/jira/browse/HARMONY-2540
> Project: Harmony
> Issue Type: Improvement
> Components: DRLVM
> Reporter: Alexander Kleymenov
> Assigned To: Alexey Varlamov
> Attachments: patch-reg-test-fwk.txt, patch-regression-testing.txt
>
>
> At present the using of DRLVM regression testing is complicated by the following reasons:
> - It does not support automated test integration: integration of new regression test requires changes in build system.
> - It does not contain the document describing the directions for developers of new regression tests.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HARMONY-2540) [drlvm][testing] Regression
Testing Infrastructure implementation.
Posted by "Alexander Kleymenov (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HARMONY-2540?page=comments#action_12461158 ]
Alexander Kleymenov commented on HARMONY-2540:
----------------------------------------------
> 1) Strict compliance with DRLVM build system is not required; enough
> said why we want to replace it (just browse dev-list archive if
> interested) and this framework could be a training ground for
> alternative approach.
I don't want my work to be your training ground.
Work out definite rules for your "alternative approach" first!
Commenting of not working lines does not seem to be right.
Although you are committer and you are deciding what is right and what is wrong.
But in the future don't tell me I didn't warn you :)
> Regarding the <select>, could you please explain
> why we need to link with libstc++ here? Anyway, there should be
> conventional alternative to that hack with <select>.
You do not provide me with information about build conventions,
why should I tell you about libstc++ ? :) (hint - try to compile cpp on *nix)
> 2) Please do not intermix functional changes with beautifications -
> let's skip wrong indentation for now.
I want my work to be of hight quality. But your extra space make its
look as a bugaboo. I can't understand your reasons to do this.
> 3) Error reporting is incomplete indeed (as it was in the original
> patch), dedicated patch is welcomed (probably in separate JIRA).
Please, don't mix JVM crash handling and test failure report. This is a
second one. What about former - let's leave it for a next discussion.
> 4) Extra test does no harm, right? Missing JIRA entry does not void this
> test, it still to be included somewhere - so let's keep it for a while
It's still in src/test/jni/nio .
It was an example taken from there. And now I don't want it to be here.
So, please, remove it from regression test suite.
Alexander
> [drlvm][testing] Regression Testing Infrastructure implementation.
> ------------------------------------------------------------------
>
> Key: HARMONY-2540
> URL: http://issues.apache.org/jira/browse/HARMONY-2540
> Project: Harmony
> Issue Type: Improvement
> Components: DRLVM
> Reporter: Alexander Kleymenov
> Assigned To: Alexey Varlamov
> Attachments: patch-reg-test-fwk.txt, patch-regression-testing.txt
>
>
> At present the using of DRLVM regression testing is complicated by the following reasons:
> - It does not support automated test integration: integration of new regression test requires changes in build system.
> - It does not contain the document describing the directions for developers of new regression tests.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Closed: (HARMONY-2540) [drlvm][testing] Regression Testing
Infrastructure implementation.
Posted by "Alexey Varlamov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HARMONY-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexey Varlamov closed HARMONY-2540.
------------------------------------
> [drlvm][testing] Regression Testing Infrastructure implementation.
> ------------------------------------------------------------------
>
> Key: HARMONY-2540
> URL: https://issues.apache.org/jira/browse/HARMONY-2540
> Project: Harmony
> Issue Type: Improvement
> Components: DRLVM
> Reporter: Alexander Kleymenov
> Assigned To: Alexey Varlamov
> Attachments: patch-reg-test-fwk.txt, patch-regression-testing.txt
>
>
> At present the using of DRLVM regression testing is complicated by the following reasons:
> - It does not support automated test integration: integration of new regression test requires changes in build system.
> - It does not contain the document describing the directions for developers of new regression tests.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HARMONY-2540) [drlvm][testing] Regression
Testing Infrastructure implementation.
Posted by "Alexey Varlamov (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HARMONY-2540?page=comments#action_12461156 ]
Alexey Varlamov commented on HARMONY-2540:
------------------------------------------
Alexander, you did not convince me yet :)
1) Strict compliance with DRLVM build system is not required; enough said why we want to replace it (just browse dev-list archive if interested) and this framework could be a training ground for alternative approach. Regarding the <select>, could you please explain why we need to link with libstc++ here? Anyway, there should be conventional alternative to that hack with <select>.
2) Please do not intermix functional changes with beautifications - let's skip wrong indentation for now.
3) Error reporting is incomplete indeed (as it was in the original patch), dedicated patch is welcomed (probably in separate JIRA).
4) Extra test does no harm, right? Missing JIRA entry does not void this test, it still to be included somewhere - so let's keep it for a while
> [drlvm][testing] Regression Testing Infrastructure implementation.
> ------------------------------------------------------------------
>
> Key: HARMONY-2540
> URL: http://issues.apache.org/jira/browse/HARMONY-2540
> Project: Harmony
> Issue Type: Improvement
> Components: DRLVM
> Reporter: Alexander Kleymenov
> Assigned To: Alexey Varlamov
> Attachments: patch-reg-test-fwk.txt, patch-regression-testing.txt
>
>
> At present the using of DRLVM regression testing is complicated by the following reasons:
> - It does not support automated test integration: integration of new regression test requires changes in build system.
> - It does not contain the document describing the directions for developers of new regression tests.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HARMONY-2540) [drlvm][testing] Regression
Testing Infrastructure implementation.
Posted by "Alexander Kleymenov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HARMONY-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12471190 ]
Alexander Kleymenov commented on HARMONY-2540:
----------------------------------------------
Looks fine after postfix at revision #504545.
Thank you,
Alexander
> [drlvm][testing] Regression Testing Infrastructure implementation.
> ------------------------------------------------------------------
>
> Key: HARMONY-2540
> URL: https://issues.apache.org/jira/browse/HARMONY-2540
> Project: Harmony
> Issue Type: Improvement
> Components: DRLVM
> Reporter: Alexander Kleymenov
> Assigned To: Alexey Varlamov
> Attachments: patch-reg-test-fwk.txt, patch-regression-testing.txt
>
>
> At present the using of DRLVM regression testing is complicated by the following reasons:
> - It does not support automated test integration: integration of new regression test requires changes in build system.
> - It does not contain the document describing the directions for developers of new regression tests.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HARMONY-2540) [drlvm][testing] Regression Testing
Infrastructure implementation.
Posted by "Alexey Varlamov (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HARMONY-2540?page=all ]
Alexey Varlamov reassigned HARMONY-2540:
----------------------------------------
Assignee: Alexey Varlamov
> [drlvm][testing] Regression Testing Infrastructure implementation.
> ------------------------------------------------------------------
>
> Key: HARMONY-2540
> URL: http://issues.apache.org/jira/browse/HARMONY-2540
> Project: Harmony
> Issue Type: Improvement
> Components: DRLVM
> Reporter: Alexander Kleymenov
> Assigned To: Alexey Varlamov
> Attachments: patch-regression-testing.txt
>
>
> At present the using of DRLVM regression testing is complicated by the following reasons:
> - It does not support automated test integration: integration of new regression test requires changes in build system.
> - It does not contain the document describing the directions for developers of new regression tests.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HARMONY-2540) [drlvm][testing] Regression
Testing Infrastructure implementation.
Posted by "Alexander Kleymenov (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HARMONY-2540?page=comments#action_12461149 ]
Alexander Kleymenov commented on HARMONY-2540:
----------------------------------------------
Alexey,
> Alexander, your latest patch basically undoes my modifications for uncoupling
> reg.test targets with super-duper meta-component-designed drlvm build system,
> and removes regression/H0000 test.
Thanks for asking. Let's go file by file:
reg.test.xml, build.xml:
------------------------
I made your changes to comply with overall DRLVM build system.
Believe me, it is better than commenting <select> element :)
This fix SHOULD be applied.
reg.test.run.xml:
-----------------
First - it fixes indentation problem made by your patch
(+1 extra space char at the beginning of the lines).
Second - "super-duper" is an error reporting system used by custom
test launchers. The framework is incomplete without it.
This fix SHOULD be applied.
> The H0000 test is a nice grindstone for the
> regression framework so I prefer to keep it fot now.
H0000:
------
At the beginning this test was provided as an example and it is not
connected with any issue report.
So it is NOT A REGRESSION TEST (by definition) and SHOULD be removed
from regression test suite.
> + <if>
> + <isset property="reg.test.custom.launched"/>
> + <then>
> + <fail message="${reg.test.failed.mark}">
> + <condition>
> + <istrue value="${reg.test.errors}"/>
> + </condition>
> + </fail>
> + </then>
> + </if>
> </sequential>
> </macrodef>
>
> BTW, the last one is a good sample to improve ant-scripting style.
> Compare to the following alternative:
> @@ -160,9 +161,10 @@
> <formatter type="brief"/>
> <formatter type="xml"/>
> </junit>
> + <fail message="${reg.test.failed.mark}" if="${reg.test.errors}"/>
I believe you know Ant well, but your suggestion does different
things from expected. I'm not happy with it (and reg testing framework
too).
So I made a conclusion that patch should be applied as it is.
Thanks,
Alexander
> [drlvm][testing] Regression Testing Infrastructure implementation.
> ------------------------------------------------------------------
>
> Key: HARMONY-2540
> URL: http://issues.apache.org/jira/browse/HARMONY-2540
> Project: Harmony
> Issue Type: Improvement
> Components: DRLVM
> Reporter: Alexander Kleymenov
> Assigned To: Alexey Varlamov
> Attachments: patch-reg-test-fwk.txt, patch-regression-testing.txt
>
>
> At present the using of DRLVM regression testing is complicated by the following reasons:
> - It does not support automated test integration: integration of new regression test requires changes in build system.
> - It does not contain the document describing the directions for developers of new regression tests.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira