You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Chris Hostetter <ho...@fucit.org> on 2010/04/29 00:57:15 UTC
Solr JUnit test methods steping on eachother?
I was out of hte loop when the changes where made the all of hte Solr test
to make then more "JUnit4-ish" but now i'm trying to add some new test
methods and finding that my new tests, even when they succeed, are causing
ohter tests to fail -- even if i make no changes to the underlying code.
Below is a patch against solr -- as you can see all it does is add a new
test method that doesn't even assert anything, it just populates the index
(my real case did more, but i trimmed the example down just to demonstrate
the problem) if you apply this patch, and then run "ant test-core
-Dtestcase=SimpleFacetsTest" you'll get a failure from the
SimpleFacetsTest.testFacetPrefixSingleValued test. Looking at the test
output, the failure seems to come from finding an extra "prefix" value in
the prefix faceting output.
As best i can figure the current setup isn't cleaning out the index
between each of the running test methods -- i'm basing this on my limited
understanding of JUnit4, and then call to "initCore" in the
"beforeClass()" method (presumably this means the core is only
initialized when the Test class is constructed, and not prior to each
test method like it use to me).
But even with that in mind, i can't make sense of why the test is failing:
my new code isn't adding anything to that field.
So WTF is going on?
1) am i understanding "beforeClass" correctly?
2) if so, then how is the current setup suppose to ensure that testAAA
doesn't leave garbase arround that screws up testBBB ?
3) even if the anwer to #2 is "we don't ensure that" then how is my new
test method screwing things up in a way that the other exsiting test
methods aren't ?
(Note: i've seen some other threads about the test parallization stuff
in the lucene-java tests having problems -- but the solr tests appear to
be run in a "junit-sequential" target, so i don't think that's the
problem)
Patch....
Index: src/test/org/apache/solr/request/SimpleFacetsTest.java
===================================================================
--- src/test/org/apache/solr/request/SimpleFacetsTest.java (revision 939066)
+++ src/test/org/apache/solr/request/SimpleFacetsTest.java (working copy)
@@ -390,6 +390,31 @@
}
@Test
+ public void testDateFacetsWithIncludeOption() {
+ final String f = "bday";
+ final String pre = "//lst[@name='facet_dates']/lst[@name='"+f+"']";
+
+ // similar to testDateFacets
+ final String ooo = "00:00:00.000Z";
+
+ assertU(adoc("id", "0", f, "1900-01-01T"+ooo));
+ assertU(commit());
+ assertU(adoc("id", "1", f, "1976-07-01T"+ooo));
+ assertU(adoc("id", "2", f, "1976-07-04T"+ooo));
+ assertU(adoc("id", "3", f, "1976-07-05T"+ooo));
+ assertU(adoc("id", "4", f, "1976-07-05T00:07:67.890Z"));
+ assertU(commit());
+ assertU(adoc("id", "5", f, "1976-07-07T"+ooo));
+ assertU(adoc("id", "6", f, "1976-07-13T"+ooo));
+ assertU(adoc("id", "7", f, "1976-07-13T00:07:67.890Z"));
+ assertU(adoc("id", "8", f, "1976-07-15T15:15:15.155Z"));
+ assertU(commit());
+ assertU(adoc("id", "9", f, "2000-01-01T"+ooo));
+ assertU(commit());
+ }
+
+
+ @Test
public void testFacetMultiValued() {
doFacetPrefix("t_s", "facet.method","enum");
doFacetPrefix("t_s", "facet.method", "enum", "facet.enum.cache.minDf", "2");
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
RE: Solr JUnit test methods steping on eachother?
Posted by Uwe Schindler <uw...@thetaphi.de>.
We not only want to reuse the core, we also want to reuse indexes. If the index is built in beforeClass you can run all tests against it. If you need another index, write another test class :-) This speeded up the tests in Lucene and Solr a lot. We now build indexes, that are global in the beforeClass and destroy them in after class (and nulling out references to make the GC happy). Tests like TestIndexWriter in Lucene, that *modify* indexes, do not build any indexes in beforeClass, as index creation is part of the test.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
> Sent: Thursday, April 29, 2010 2:10 AM
> To: Lucene Dev
> Subject: Re: Solr JUnit test methods steping on eachother?
>
>
> : 1) am i understanding "beforeClass" correctly?
> : 2) if so, then how is the current setup suppose to ensure that
> testAAA
> : doesn't leave garbase arround that screws up testBBB ?
>
> After skimming the JUnit annotation docs and experimenting, it looks
> like
> the SolrCore is definitely getting reused, and i was able to clean up
> the existing index by adding this to the test class...
>
> @After
> public void after() throws Exception {
> assertU(delQ("*:*"));
> assertU(commit());
> }
>
> ...if reusing the SolreCore for each "test" method was deliberate, then
> shouldn't something like this be in the base class?
>
> I still can't make sense of why this particular patch caused this
> particular failure -- none of hte docs i added had the field
> facet.prefix was being used on...
>
> : 3) even if the anwer to #2 is "we don't ensure that" then how is my
> new test
> : method screwing things up in a way that the other exsiting test
> methods aren't
> : ?
>
> ...
>
> : Patch....
> :
> : Index: src/test/org/apache/solr/request/SimpleFacetsTest.java
> : ===================================================================
> : --- src/test/org/apache/solr/request/SimpleFacetsTest.java
> (revision
> : 939066)
> : +++ src/test/org/apache/solr/request/SimpleFacetsTest.java
> (working copy)
> : @@ -390,6 +390,31 @@
> : }
> :
> : @Test
> : + public void testDateFacetsWithIncludeOption() {
> : + final String f = "bday";
> : + final String pre =
> "//lst[@name='facet_dates']/lst[@name='"+f+"']";
> : +
> : + // similar to testDateFacets
> : + final String ooo = "00:00:00.000Z";
> : +
> : + assertU(adoc("id", "0", f, "1900-01-01T"+ooo));
> : + assertU(commit());
> : + assertU(adoc("id", "1", f, "1976-07-01T"+ooo));
> : + assertU(adoc("id", "2", f, "1976-07-04T"+ooo));
> : + assertU(adoc("id", "3", f, "1976-07-05T"+ooo));
> : + assertU(adoc("id", "4", f, "1976-07-05T00:07:67.890Z"));
> : + assertU(commit());
> : + assertU(adoc("id", "5", f, "1976-07-07T"+ooo));
> : + assertU(adoc("id", "6", f, "1976-07-13T"+ooo));
> : + assertU(adoc("id", "7", f, "1976-07-13T00:07:67.890Z"));
> : + assertU(adoc("id", "8", f, "1976-07-15T15:15:15.155Z"));
> : + assertU(commit());
> : + assertU(adoc("id", "9", f, "2000-01-01T"+ooo));
> : + assertU(commit());
> : + }
> : +
> : +
> : + @Test
> : public void testFacetMultiValued() {
> : doFacetPrefix("t_s", "facet.method","enum");
> : doFacetPrefix("t_s", "facet.method", "enum",
> "facet.enum.cache.minDf",
> : "2");
> :
> :
> : ---------------------------------------------------------------------
> : To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> : For additional commands, e-mail: dev-help@lucene.apache.org
> :
>
>
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Solr JUnit test methods steping on eachother?
Posted by Chris Hostetter <ho...@fucit.org>.
: 1) am i understanding "beforeClass" correctly?
: 2) if so, then how is the current setup suppose to ensure that testAAA
: doesn't leave garbase arround that screws up testBBB ?
After skimming the JUnit annotation docs and experimenting, it looks like
the SolrCore is definitely getting reused, and i was able to clean up
the existing index by adding this to the test class...
@After
public void after() throws Exception {
assertU(delQ("*:*"));
assertU(commit());
}
...if reusing the SolreCore for each "test" method was deliberate, then
shouldn't something like this be in the base class?
I still can't make sense of why this particular patch caused this
particular failure -- none of hte docs i added had the field
facet.prefix was being used on...
: 3) even if the anwer to #2 is "we don't ensure that" then how is my new test
: method screwing things up in a way that the other exsiting test methods aren't
: ?
...
: Patch....
:
: Index: src/test/org/apache/solr/request/SimpleFacetsTest.java
: ===================================================================
: --- src/test/org/apache/solr/request/SimpleFacetsTest.java (revision
: 939066)
: +++ src/test/org/apache/solr/request/SimpleFacetsTest.java (working copy)
: @@ -390,6 +390,31 @@
: }
:
: @Test
: + public void testDateFacetsWithIncludeOption() {
: + final String f = "bday";
: + final String pre = "//lst[@name='facet_dates']/lst[@name='"+f+"']";
: +
: + // similar to testDateFacets
: + final String ooo = "00:00:00.000Z";
: +
: + assertU(adoc("id", "0", f, "1900-01-01T"+ooo));
: + assertU(commit());
: + assertU(adoc("id", "1", f, "1976-07-01T"+ooo));
: + assertU(adoc("id", "2", f, "1976-07-04T"+ooo));
: + assertU(adoc("id", "3", f, "1976-07-05T"+ooo));
: + assertU(adoc("id", "4", f, "1976-07-05T00:07:67.890Z"));
: + assertU(commit());
: + assertU(adoc("id", "5", f, "1976-07-07T"+ooo));
: + assertU(adoc("id", "6", f, "1976-07-13T"+ooo));
: + assertU(adoc("id", "7", f, "1976-07-13T00:07:67.890Z"));
: + assertU(adoc("id", "8", f, "1976-07-15T15:15:15.155Z"));
: + assertU(commit());
: + assertU(adoc("id", "9", f, "2000-01-01T"+ooo));
: + assertU(commit());
: + }
: +
: +
: + @Test
: public void testFacetMultiValued() {
: doFacetPrefix("t_s", "facet.method","enum");
: doFacetPrefix("t_s", "facet.method", "enum", "facet.enum.cache.minDf",
: "2");
:
:
: ---------------------------------------------------------------------
: To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
: For additional commands, e-mail: dev-help@lucene.apache.org
:
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Solr JUnit test methods steping on eachother?
Posted by Chris Hostetter <ho...@fucit.org>.
: > Hmmm... except they were totally independent before.
:
: Conversion to the new style is not automatic - older style tests
: (those that extend TestCase) run as they always did.
Hmmm ... i'm sorry, my mistake, i spot checked a bunch of tests and
thought that everything which had previously subclassed
AbstractSolrTestCase was now subclassing -- being more thorough i realize
that's not true.
: There was a lot of work put into making them pass for those that were
: upgraded to the new style.
Right, of course ... when the change was made a bunch of tests would have
started breaking and then those tests were modified to either make lss
general assumptions or clean up the data that was confusing them. that
makes total sense.
I'm still at a total loss to understand how that simple example test
method from my patch would have caused a problem in the facet.prefix test
since it didn't even use the same fields. my best guess is that the new
index size caused segment merges to happen at differnet points causing
differnet deleted docs to be (or not be) expunged when the test ran. If
that's the case hopefully your new "randomized" document creation code
will also uncover this problem.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Solr JUnit test methods steping on eachother?
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Wed, Apr 28, 2010 at 8:16 PM, Chris Hostetter
<ho...@fucit.org> wrote:
> : That's extra time, and some test methods may rely on a persistent
> : index across multiple test methods. The move to Junit4 was not back
> : compatible - test methods are not independent.
>
> Hmmm... except they were totally independent before.
Conversion to the new style is not automatic - older style tests
(those that extend TestCase) run as they always did.
> I'm actually amazed that our tests pass at all
There was a lot of work put into making them pass for those that were
upgraded to the new style.
-Yonik
Apache Lucene Eurocon 2010
18-21 May 2010 | Prague
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Solr JUnit test methods steping on eachother?
Posted by Chris Hostetter <ho...@fucit.org>.
: That's extra time, and some test methods may rely on a persistent
: index across multiple test methods. The move to Junit4 was not back
: compatible - test methods are not independent.
Hmmm... except they were totally independent before.
I'm actually amazed that our tests pass at all -- if every method leaves
stuff in the index, then i would have expected lots of tests to fail
because document counts on searches don't match what they expect. I
suspect that we've just gotten really lucky: within a given test class,
either the methods are only looking for hte data they expect, or they are
overwriting the docs added by methods that ran earlier (by reusing
uniqueKey ids)
: We could certainly have a little utility method that could be called,
: but I'm not sure it should be called by default.
but if it's not on by default then there is nothing to clean up data when
another method fails which could cause other unrelated tests to fail: i
can write a test method that adds docs, searches them, and then deletes
them; but if something breaks the middle part, the deletes never happen
and the left over data could cause a false negative in the next test.
This could be an argument for more distinct Test classes containing fewer
methods -- but then we're right back to re-initing the core very
frequently, which seems like hte opposite of what you want.
Personally, i liked knowing i had a garunteed brand new core before every
test method, but if we're oing to reuse SolrCores then at the very least
we should have an *empty* SolrCore before the test starts.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Solr JUnit test methods steping on eachother?
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Wed, Apr 28, 2010 at 7:59 PM, Chris Hostetter
<ho...@fucit.org> wrote:
> I'm still left wondering why (if we're going to reuse teh SolrCore over
> and over) we don't have a common method in the test base class thta clears
> out the index after each test.
That's extra time, and some test methods may rely on a persistent
index across multiple test methods. The move to Junit4 was not back
compatible - test methods are not independent.
We could certainly have a little utility method that could be called,
but I'm not sure it should be called by default.
-Yonik
http://www.lucidimagination.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Solr JUnit test methods steping on eachother?
Posted by Chris Hostetter <ho...@fucit.org>.
: Yes, on purpose.
: I've actually restructured how this tests works a lot more in
: https://issues.apache.org/jira/browse/SOLR-1875 - let me commit that
: before proceeding with what you are trying to add.
:
: I actually index all of the documents only once, but in a random order
: and with random commits. This exercises many more edge cases such as
: segments w/o the particular field you are faceting on, and fields
: after (in term-enum sense) the field you are faceting on.
ok .. except that none of hte code/tests i'm adding has anything to do
with field/prefix faceting -- i'm looking ta date faceting, and the docs
i'm adding in my new date faceting tests don't have any fields in common
with the prefix test that started breaking.
I'm still left wondering why (if we're going to reuse teh SolrCore over
and over) we don't have a common method in the test base class thta clears
out the index after each test.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Solr JUnit test methods steping on eachother?
Posted by Yonik Seeley <ys...@gmail.com>.
On Wed, Apr 28, 2010 at 6:57 PM, Chris Hostetter
<ho...@fucit.org> wrote:
> As best i can figure the current setup isn't cleaning out the index between
> each of the running test methods
Yes, on purpose.
I've actually restructured how this tests works a lot more in
https://issues.apache.org/jira/browse/SOLR-1875 - let me commit that
before proceeding with what you are trying to add.
I actually index all of the documents only once, but in a random order
and with random commits. This exercises many more edge cases such as
segments w/o the particular field you are faceting on, and fields
after (in term-enum sense) the field you are faceting on.
-Yonik
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org