You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Grant Ingersoll <gs...@apache.org> on 2010/10/14 19:58:32 UTC
Speaking of tests
Anyone know how to print out, at the end of a long run of tests, the names of the actual tests that fail to the console? It's a pain to scroll through all the output to find the line and grepping through the reports is OK, but it would just be so much easier if it was summarized right there at the end of the run.
Just want something to the effect of the <fail> tag there at the end of the target to also print the names of the classes that failed.
-Grant
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Speaking of tests
Posted by Chris Hostetter <ho...@fucit.org>.
: +1, this is the place to do it. you can get the test stats in
: endTestSuite(JUnitTest suite)
: and append it to a file (set from a sysprop).
:
: then later in the build, when the "tests.failed" sysprop is set, we
: cat the file.
I hadn't even though of that ... i was assuming we'd need to implement the
"BuildListener" interface and use the "taskFinished" method (like
FailureRecorder does) to write the info to the console.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Speaking of tests
Posted by Chris Hostetter <ho...@fucit.org>.
: i just added a patch to SOLR-2002 so that we only check the
: "tests.failed" once so you wont see it a ton of times... or the BUILD
: FAILED! over and over and over again.
I actually thought that was intentional ... the a test didn't just "FAIL"
several tests "FAILED FAILED FAILED!!!!!!"
(kind of like: you aren't just "under arrest" for a misdomenier, you are
"UNDER ARREST FOR A FEDERAL FELONY!!!")
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Speaking of tests
Posted by Robert Muir <rc...@gmail.com>.
On Thu, Oct 14, 2010 at 9:54 PM, Chris Hostetter
<ho...@fucit.org> wrote:
> It might be more reliable to modify our recently add
> LuceneJUnitResultFormatter to do tis aggregation as it collects failure
> info ... i'm not seeing any sort of "tear down" or "close" method though,
> so i'm not sure if it's really possible.
>
+1, this is the place to do it. you can get the test stats in
endTestSuite(JUnitTest suite)
and append it to a file (set from a sysprop).
then later in the build, when the "tests.failed" sysprop is set, we
cat the file.
i just added a patch to SOLR-2002 so that we only check the
"tests.failed" once so you wont see it a ton of times... or the BUILD
FAILED! over and over and over again.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Speaking of tests
Posted by Chris Hostetter <ho...@fucit.org>.
: In Junit4, there's a TestWatchman class that has a failed method that's
: called whenever a test fails. It *seems* like it would be possible to
: gather them all up and print them at the end, but I confess I have no
: clue how, and don't have time to look now.
interesting...
the intent of TestWatchmen seems to be so that individual test classes can
configure certain things to happen when *any* of the tests fail (or have
some interesting condition happen) ... we could put that in our JUnit base
class ... but ...
It might be more reliable to modify our recently add
LuceneJUnitResultFormatter to do tis aggregation as it collects failure
info ... i'm not seeing any sort of "tear down" or "close" method though,
so i'm not sure if it's really possible.
Hmmm...
I do note that in Ant 1.8.0 there is a "failure" formatter that looks kind
of cool ... it's purpose is to generate a new TestCase that contains calls
to all of the tests that failed, but the lifecyle concepts could be
borrowed to generate the same kind of info in a summary report...
http://ant.apache.org/manual/Tasks/junit.html
http://svn.apache.org/repos/asf/ant/core/trunk/src/main/org/apache/tools/ant/taskdefs/optional/junit/FailureRecorder.java
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Speaking of tests
Posted by Erick Erickson <er...@gmail.com>.
In Junit4, there's a TestWatchman class that has a failed method that's
called whenever a test fails. It *seems* like it would be possible to
gather them all up and print them at the end, but I confess I have no
clue how, and don't have time to look now.
Not much help, but...
Erick
On Thu, Oct 14, 2010 at 6:35 PM, Chris Hostetter
<ho...@fucit.org>wrote:
>
> : Anyone know how to print out, at the end of a long run of tests, the
> : names of the actual tests that fail to the console? It's a pain to
>
> I typically use...
>
> grep -rL 'errors="0" failures="0"' test_output_dir
>
> (where test_output_dir depends on which set of tests failed)
>
> however two recent changes seem to have made this problematic...
>
> 1) all of the "temp" files for tetss now seem to live in the
> test_output_dir ... i know thta the threading issues made it important to
> start splitting up the working dirs for each test, but i'm relaly not sure
> while they really need to be in the same dir as the output files .. in any
> case adding a "*.xml" was my work arround when that start happening.
>
> 2) Solr tests recently started using the "Plain text" formatter by
> default instead of the xml formatter. this seems like a mistake to me,
> but i'm not certain.
>
>
> -Hoss
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: Speaking of tests
Posted by Chris Hostetter <ho...@fucit.org>.
: Anyone know how to print out, at the end of a long run of tests, the
: names of the actual tests that fail to the console? It's a pain to
I typically use...
grep -rL 'errors="0" failures="0"' test_output_dir
(where test_output_dir depends on which set of tests failed)
however two recent changes seem to have made this problematic...
1) all of the "temp" files for tetss now seem to live in the
test_output_dir ... i know thta the threading issues made it important to
start splitting up the working dirs for each test, but i'm relaly not sure
while they really need to be in the same dir as the output files .. in any
case adding a "*.xml" was my work arround when that start happening.
2) Solr tests recently started using the "Plain text" formatter by
default instead of the xml formatter. this seems like a mistake to me,
but i'm not certain.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Speaking of tests
Posted by Shai Erera <se...@gmail.com>.
What we use to do is run 'ant test' with a redirection of standard output to
a file. That way, the errors stand out - they appear in the console, and the
rest of the test output goes to the file.
Don't know if that's what you want, but it sure makes it much easier to spot
the failing tests.
Shai
On Thu, Oct 14, 2010 at 9:16 PM, Mark Miller <ma...@gmail.com> wrote:
> We could, at the least, make
>
> TEST org.apache.solr.analysis.TestTypeAsFieldFilterFactory FAILED
>
> stand out a bit more - prefix with a bunch of ***** or whatever.
>
> It would be nice if the red (in eclipse) error stream printed the TEST
> FAILED correctly, but I don't think that would be very easy to sync up
> with std out...
>
> - Mark
>
> On 10/14/10 3:00 PM, Robert Muir wrote:
> > On Thu, Oct 14, 2010 at 2:28 PM, Grant Ingersoll <gs...@apache.org>
> wrote:
> >
> >> but, I just want a simple print out of the class names, as in:
> >>
> >> Tests failed:
> >> FooTest.testFoo
> >> BarTest.testBar
> >>
> >> Patches welcome, I suppose!
> >>
> >
> > maybe open an issue?
> > I think its possible we can improve this.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: dev-help@lucene.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: Speaking of tests
Posted by Mark Miller <ma...@gmail.com>.
We could, at the least, make
TEST org.apache.solr.analysis.TestTypeAsFieldFilterFactory FAILED
stand out a bit more - prefix with a bunch of ***** or whatever.
It would be nice if the red (in eclipse) error stream printed the TEST
FAILED correctly, but I don't think that would be very easy to sync up
with std out...
- Mark
On 10/14/10 3:00 PM, Robert Muir wrote:
> On Thu, Oct 14, 2010 at 2:28 PM, Grant Ingersoll <gs...@apache.org> wrote:
>
>> but, I just want a simple print out of the class names, as in:
>>
>> Tests failed:
>> FooTest.testFoo
>> BarTest.testBar
>>
>> Patches welcome, I suppose!
>>
>
> maybe open an issue?
> I think its possible we can improve this.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Speaking of tests
Posted by Robert Muir <rc...@gmail.com>.
On Thu, Oct 14, 2010 at 2:28 PM, Grant Ingersoll <gs...@apache.org> wrote:
> but, I just want a simple print out of the class names, as in:
>
> Tests failed:
> FooTest.testFoo
> BarTest.testBar
>
> Patches welcome, I suppose!
>
maybe open an issue?
I think its possible we can improve this.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Speaking of tests
Posted by Grant Ingersoll <gs...@apache.org>.
On Oct 14, 2010, at 2:18 PM, Erik Hatcher wrote:
> The JUnit report task will generate a nice HTML version that illustrates with red what tests failed and you can easily drill into it. We have that wired into our build, right?
Yeah, I can look at the reports, etc.
>
> Granted, it'd be nice to see a summary at the end of a JUnit run in the console, but I don't know of a way to make that happen.
but, I just want a simple print out of the class names, as in:
Tests failed:
FooTest.testFoo
BarTest.testBar
Patches welcome, I suppose!
>
> Erik
>
> On Oct 14, 2010, at 13:58 , Grant Ingersoll wrote:
>
>> Anyone know how to print out, at the end of a long run of tests, the names of the actual tests that fail to the console? It's a pain to scroll through all the output to find the line and grepping through the reports is OK, but it would just be so much easier if it was summarized right there at the end of the run.
>>
>> Just want something to the effect of the <fail> tag there at the end of the target to also print the names of the classes that failed.
>>
>> -Grant
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Speaking of tests
Posted by Erik Hatcher <er...@gmail.com>.
The JUnit report task will generate a nice HTML version that illustrates with red what tests failed and you can easily drill into it. We have that wired into our build, right?
Granted, it'd be nice to see a summary at the end of a JUnit run in the console, but I don't know of a way to make that happen.
Erik
On Oct 14, 2010, at 13:58 , Grant Ingersoll wrote:
> Anyone know how to print out, at the end of a long run of tests, the names of the actual tests that fail to the console? It's a pain to scroll through all the output to find the line and grepping through the reports is OK, but it would just be so much easier if it was summarized right there at the end of the run.
>
> Just want something to the effect of the <fail> tag there at the end of the target to also print the names of the classes that failed.
>
> -Grant
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org