[Ant Wiki] Update of "Proposals/EnhancedTestReports" by SteveLoughran

[Apache Wiki <wi...@apache.org>]

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Ant Wiki" for change notification.

The following page has been changed by SteveLoughran:
http://wiki.apache.org/ant/Proposals/EnhancedTestReports

The comment on the change is:
more thoughts; spelling

------------------------------------------------------------------------------
  For Ant we'd be looking at ant1.8.0 and any 1.7.2 release for these changes; gives us time to test with all the CI servers. We could do a junit4 test runner with early output. Testng are on a faster release cycle. 
  
   1. Start wiki, discussions on various lists (ant-user, testng-user)
+  1. collect test data: XML files from all the tools that currently generate the format
   1. define first relaxng schema
-  1. set up vms (EC2? Apache VM farm?) with the various CI servers
+  1. build functional test systems: Public EC2 image with all the OSS CI tools installed; Private VMWare images with Bamboo, TeamCity. 
   1. add new <junit> reporter, test its output works with existing consumers
   1. add antunit support (pull out description; if/unless => Skipped)
   1. evolve XSL stylesheets
@@ -95, +96 @@

  
   * Retain the current stdout+stderr logs, but also allow people to bind to custom log4j/commons-logging/java.util.logging back ends that grab the raw events and log them as structured XML events (host,process,thread,level,timestamp,text). These would be renderable at different levels
   * XSL style sheets to merge output, ordering by clock (received time) and displaying log messages from different machines/levels in different colour
-  * easy turn on/off display of different levels
+  * easy turn on/off display of different levels in viewers; maybe test runner.
  
  === Improved Host information ===
  
@@ -103, +104 @@

   * wall time and time zone
   * Ant-determined OS (using standard names)
   * environment variables on java5+
+  * Java5 JMX info on JVM state pre- and post- run -eg, memory footprint of every test. 
- The risk here is that we could massively increase XML file size by duplicating all this stuff for every test. We may only need this per test suite.
+ The risk here is that we could massively increase XML file size by duplicating all this stuff for every test. We may only need some stuff per test suite.
  
  === Improved Fault Information ===
  
@@ -126, +128 @@

  
  DanFabulich: Many popular tools parse exceptions directly; they're already reasonably structured, even if they aren't really XML.
  
- SteveLoughran: True. But I'm thinking about better XSL support. Also the standard exception dumps dont scale well to deeply chained exceptions; gets hard to see what is going on.
+ SteveLoughran: True. But I'm thinking about better XSL support. Also the standard exception dumps don't scale well to deeply chained exceptions; gets hard to see what is going on.
  
  
  == Open ended failure categories ==
@@ -142, +144 @@

  * Error - unable to determine pass or fail, which includes timeout
  * Skipped
  * In Progress, aka not-yet-finished
+ 
+ SteveLoughran: I like failure itself to be categorised. So every state can be a pass state, a warning or a failure. Warnings could include "passing, but took 50% longer than usual". Failures can include "tests passed but outside allowed time" "tests passed but memory consumption was over the limit", as well as simple "some assertion failed". 
  
  == Add partial-failure, not-yet-finished as test outcomes ==
  
@@ -175, +179 @@

  Arguments against
   1. if it becomes a wire format, it becomes even less flexible
   1. CI tools are better of with their own custom listeners (they get to stop the run too :)
-  1. We dont want to endorse JAXB (brittle) or DOM (painful), but XOM is off limits to apache projects (LGPL), and we dont want to intro new dependencies to ant either.
+  1. We don't want to endorse JAXB (brittle) or DOM (painful), but XOM is off limits to apache projects (LGPL), and we don't want to intro new dependencies to ant either.
  
  This may be something for things other than Ant to consider.
  

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org