Re: Testing logfile statistics

[Sh...@lotus.com]

Hmm - which statistics are you talking about?  Which Xalan version/OS, etc.
and could you send me a sample of your logFile?

(kidding  8-)

Below find an attached ConformanceTest.xml that (to me) looks like the
output I'm expecting from the testing environment.  Don't forget that your
loggingLevel (from 0=very little, 40ish-50ish= a bunch, 99=a lot!)
determines both how much data actually gets put into the logFile in the
first place, as well as how much gets displayed in the .html report created
by viewResults.xsl.

What I should really do is make the default loggingLevel when running tests
be 99, and have viewResults.xsl default to 40.  That way all results will
be captured when the test runs into the .xml logFile, but when you look at
the .html report, you'll only see the highlights.

(See attached file: ConformanceTest.xml)
Which statistic elements are you looking for?  <statistic> is a general
purpose element that I use for a couple of different purposes:

<statistic level="60" desc="CSrt;Conformance Test of:
numbering"><longval>975950367933</longval><doubleval>0.0</doubleval></statistic>
Is used at start/end of testcase and testfile elements to get simple timing
for the whole test from currentTimeMillis()

<statistic level="40" desc="Dir average time to process files in this
dir"><longval>268</longval><doubleval>0.0</doubleval></statistic>
Is output by the XSLDirectoryIterator and subclass test drivers, to give
general performance numbers over a large number of .xsl files

<statistic level="60" desc
="passCount[CHECKS]"><longval>81</longval><doubleval>0.0</doubleval></statistic>
Is a sample 'mini-report' the Reporter dumps out giving a big-picture of
how many tests passed/failed/whatever.

Also: there's a chance that XSLDirectoryIterator  isn't consistent about
how it logs timing info for the error directories since when I added the
processFileList() stuff I may not have updated the error directories the
same way.  This is still a @todo for me (or someone): to update the whole
iteration process to be 1) discover all tests in X location to run and
build list, 2) processFileList(), 3) iterate into SomeFilenameFilter type
of subdirectories and repeat (namely from conf/dirname into
conf/dirname/err for our current conformancetest).  That will greatly
simplify the code in that file, and make sure statistics are reported the
same way for all directories.

- Shane


---------------------- Forwarded by Shane_Curcuru@lotus.com 12/04/2000
01:01 PM ---------------------------
Shane -

When using the test package and writing output to a logfile, I noticed
that the statistic element is only written after certain tests and these
all seem to be in the error directories.  Can you check on this and see
if you're getting the statistic element written out after each test
directory is processed?

Gray

Re: Testing logfile statistics

[Gary L Peskin <ga...@firstech.com>]

Gary L Peskin wrote:
> Shane_Curcuru@lotus.com wrote:
> > Hmm - which statistics are you talking about?  Which Xalan version/OS, etc.
> These happen at the end of the <testcase> element.  As I said, I'd
> expect these elements to show up after every directory is completed (ie
> at the end of every <testcase>) but they only show up sometimes.  In
> your example logfile, you have two testcases with both of these elements
> (and the level="60" one) and the end of each testcase.
> 
> I'll try to track this down myself and see what's happening and I'll let
> you know.

Shane --

I've found the source of the discrepancy.  The behavior of the test
package is fine with me.  I just wanted to understand it.

For "normal" (ie non-err) directories, the statistics are issued only if

perfLogging=true

This is at line 383 of XSLDirectoryIterator.java:

                    // Only log these stats if perfLogging is set to
true
                    String tmp =
                        testProps.getProperty(Reporter.OPT_PERFLOGGING);

                    if ((tmp != null) && tmp.equalsIgnoreCase("true"))

However, this code is not present in the test of whether or not to emit
the statistic element for the err directories.  It would go in at line
453.  These are both in processTestDir().

Further, the statistic element is produced for err directories only if
at least one of the tests did -not- throw an exception.  If all of the
tests threw an exception, the element is not produced.

Not sure if this is the behavior you wanted but this is the way that I
see it working.

Gary

Re: Testing logfile statistics

[Gary L Peskin <ga...@firstech.com>]

Shane_Curcuru@lotus.com wrote:
> Hmm - which statistics are you talking about?  Which Xalan version/OS, etc.

Shane --  I'm using XalanJ version 2 on NT with IBM's JDK 1.3.  My
properties file (ConformanceTest.prop) specifies only the following
entries:
inputDir=...
goldDir=...
outputDir=...
logFile=...

So, I'm running all of the conformance tests in all of the various
directories.

The <statistic> elements that I'm referring to are
<statistic level="40" desc="Dir average time to process files in this
dir">
and
<statistic level="40" desc="Running average time of dirs so far">

These happen at the end of the <testcase> element.  As I said, I'd
expect these elements to show up after every directory is completed (ie
at the end of every <testcase>) but they only show up sometimes.  In
your example logfile, you have two testcases with both of these elements
(and the level="60" one) and the end of each testcase.

I'll try to track this down myself and see what's happening and I'll let
you know.

Gary