You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Erick Erickson <er...@gmail.com> on 2018/01/11 17:48:59 UTC
What are the expectations for cleanup SolrCloud tests when
tests.iters is specified?
The recent failures in TimeRoutedAliasUpdateProcessorTest. are a result of
the changes to SOLR-11218. The initial fix is easy, but while fixing it I
saw a comment in there about cleaning up when tests.iters is specified > 1.
The assumption was that everything would be cleaned up between runs doesn't
appear to be true for SolrCloud tests. I think one of two things is
happening:
1> collections (and perhaps aliases) are simply not cleaned up
2> there is a timing issue, we have waitForCollectionToDisappear in test
code after all.
My question is what are the expectations? It's pretty ugly to require each
test to clean up all changes to ZooKeeper or else fail when test.iters is
set. Plus it violates the expectation that running one test doesn't affect
the other test, even if it's the same test over and over.
So if we think it's an issue with the framework, any clues where to look
would be appreciated.
Re: What are the expectations for cleanup SolrCloud tests when
tests.iters is specified?
Posted by Robert Muir <rc...@gmail.com>.
On Mon, Jan 15, 2018 at 10:56 AM, Jason Gerlowski <ge...@gmail.com> wrote:
> It sounds like the recommendation in this thread is to _always_ use
> "ant beast" instead of "tests.iters". Is there _any_ case where
> "tests.iters" should be preferred? If not, should we remove support
> for "tests.iters" to remove any ambiguity? (Especially since this has
> come up on the list a few times...)
tests.iters is significantly faster if your test is really more of a
unit test: for example it doesn't suffer the overhead of launching a
new JVM for each iteration.
but because it shortcuts all this stuff, its also not as thorough, and
won't run the test with various class-level fixtures (different
codecs/locales/...).
So you can run your test with a 100 million iterations and then commit
it, only for it to fail instantly in jenkins for those reasons.
Personally i don't want to waste-time playing whack-a-mole like that:
if i want to beast a test i want to really beast it, so I only use ant
beast, and pretend like tests.iters doesn't exist. The last time we
brought this up I think its survival was justified by the fact that
you can combine it with ant beast to be really efficient (which is
indeed true), but its just too fragile for my tastes.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: What are the expectations for cleanup SolrCloud tests when
tests.iters is specified?
Posted by Jason Gerlowski <ge...@gmail.com>.
Ok, good clarification guys. Thanks!
On Mon, Jan 15, 2018 at 11:41 AM, Robert Muir <rc...@gmail.com> wrote:
> just to be clear, as i mentioned: tests.iters should run
> setup/teardown just fine, thats not a problem. There is no issue with
> tests.iters setup/teardown/Before/After.
>
> but its only going to instantiate your *class* a single time and run
> *class*-level stuff (e.g. BeforeClass/AfterClass) a single time.
>
>
> On Mon, Jan 15, 2018 at 11:35 AM, Erick Erickson
> <er...@gmail.com> wrote:
>> Any test that doesn't really do anything in setup/teardown seems like a
>> good candidate for using tests.iters, which will be much faster.
>>
>> For SolrCloud level tests beasting is becoming my first choice.
>>
>> For a lot of the Lucene level code tests.iters makes a lot of sense so
>> it's too useful to remove.
>>
>> Best,
>> Erick
>>
>>
>> On Mon, Jan 15, 2018 at 7:56 AM, Jason Gerlowski <ge...@gmail.com> wrote:
>>> It sounds like the recommendation in this thread is to _always_ use
>>> "ant beast" instead of "tests.iters". Is there _any_ case where
>>> "tests.iters" should be preferred? If not, should we remove support
>>> for "tests.iters" to remove any ambiguity? (Especially since this has
>>> come up on the list a few times...)
>>>
>>> On Fri, Jan 12, 2018 at 11:14 AM, Erick Erickson
>>> <er...@gmail.com> wrote:
>>>> Well, since I'm in there anyway I'll include the note in the patch. At
>>>> least that'll alert people to dig deeper.
>>>>
>>>> On Thu, Jan 11, 2018 at 8:34 PM, David Smiley <da...@gmail.com> wrote:
>>>>> Yeah thanks guys -- beast it is.
>>>>>
>>>>> I wonder if we should not document tests.iters (a bit more expert), or add a
>>>>> warning to it in the help output saying something like: NOTE: some tests are
>>>>> incompatible because BeforeClass/AfterClass isn't performed inbetween. Try
>>>>> beast.iters instead.
>>>>>
>>>>> On Thu, Jan 11, 2018 at 8:39 PM Erick Erickson <er...@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Ok, thanks both. That makes a lot of sense. I'll just use beasting for
>>>>>> most anything SolrCloud related.
>>>>>>
>>>>>>
>>>>>> On Thu, Jan 11, 2018 at 4:56 PM, Chris Hostetter
>>>>>> <ho...@fucit.org> wrote:
>>>>>>>
>>>>>>> : (I had left the comment in question)
>>>>>>> : I think a test shouldn't have to explicitly clean up after itself,
>>>>>>> except
>>>>>>> : perhaps intra-method as-needed; test-infrastructure should do the class
>>>>>>> : (test suite).
>>>>>>>
>>>>>>> All test code should always be expected to clean up their messes at
>>>>>>> whatever "ending" stage corrisponds with the stage where the mess was
>>>>>>> made.
>>>>>>>
>>>>>>> how the mess is cleaned up, and wether infrastructure/scaffolding code
>>>>>>> helps do that dpeends on the specifics of the infrastucture/scaffolding
>>>>>>> in
>>>>>>> question -- if you make a mess in a test method that the general purpose
>>>>>>> infrastructure doesn't expect, then the burden is on you
>>>>>>> to add the level of infrastructure (either in your specific test class,
>>>>>>> or
>>>>>>> in a new abstract base class depending on how you think it might be
>>>>>>> re-usable) to do so.
>>>>>>>
>>>>>>> In the abstract: Assume AbstractParentTest class that creates some
>>>>>>> "parentMess" in @BeforeClass, and deletes "parentMess" in an
>>>>>>> @AfterClass....
>>>>>>>
>>>>>>> 1) if you want all of your tests methods to have access to a
>>>>>>> shiny new/unique instance of "childMess" in every test method, then
>>>>>>> burden
>>>>>>> is on you to create/destroy childMess in your own @Before and @After
>>>>>>> methods
>>>>>>>
>>>>>>> 2) If you want test methods that are going to mutate "parentMess" then
>>>>>>> the
>>>>>>> burden is on you to ensure (ideally via @After methods that "do the right
>>>>>>> thing" even if the test method fails) that "parentMess" is correctly
>>>>>>> reset
>>>>>>> so that all the test methods depending on "parentMess" can run in any
>>>>>>> order (or run multiple times in a single instance) ... either that, or
>>>>>>> you
>>>>>>> shouldn't use AbstractParentTest -- you should create/destroy
>>>>>>> a "parentMess" instance yourself in your @Before & @After methods
>>>>>>>
>>>>>>> Concretely...
>>>>>>>
>>>>>>> : > The assumption was that everything would be cleaned up between runs
>>>>>>> : > doesn't appear to be true for SolrCloud tests. I think one of two
>>>>>>> things is
>>>>>>> : > happening:
>>>>>>> : >
>>>>>>> : > 1> collections (and perhaps aliases) are simply not cleaned up
>>>>>>> : >
>>>>>>> : > 2> there is a timing issue, we have waitForCollectionToDisappear in
>>>>>>> test
>>>>>>> : > code after all.
>>>>>>>
>>>>>>> ...these are vague statements ("everything", "SolrCloud tests", "not
>>>>>>> cleaned up") and not being intimately familiar with the test class in
>>>>>>> question it's not clear exactly is happening or what expectations various
>>>>>>> people have -- BUT -- assuming this is in regards to
>>>>>>> SolrCloudTestCase, that base class has very explicit docs about
>>>>>>> how it's intended to be used: you are expected to configure & init a
>>>>>>> MiniSolrCloudCluster instance in an @BeforeClass method -- it has helper
>>>>>>> code for this -- and that cluster lives for the lifespan of the class at
>>>>>>> which point an @AfterClass in SolrCloudTestCase will ensure it gets torn
>>>>>>> down.
>>>>>>>
>>>>>>> Tests which subclass SolrCloudTestCase should be initializing the cluster
>>>>>>> only in @BeforeClass. Most tests should only be creating collections in
>>>>>>> @BeforeClass -- allthough you are certainly free to do things like
>>>>>>> create/destroy collections on a per test method basis in @Before/@After
>>>>>>> methods if you have a need for that sort of thing.
>>>>>>>
>>>>>>> If that's not the lifecycle you want -- if you want a lifecycle where
>>>>>>> ever
>>>>>>> individual test method gets it's own pristine new MiniSolrCloudCluster
>>>>>>> instance w/o any pre-existing collections, then you shouldn't use
>>>>>>> SolrCloudTestCase -- you should just create/destroy
>>>>>>> unique MiniSolrCloudCluster instances in your own @Before/@After methods.
>>>>>>>
>>>>>>>
>>>>>>> Bottom Line: there is no one size fits all test scaffolding -- not when
>>>>>>> we
>>>>>>> have some tests classes where we want to create a collection once, fill
>>>>>>> it
>>>>>>> with lots of docs, and then re-use it in 100s of test methods, but other
>>>>>>> classes want to test the very operation of creating/deleting collections.
>>>>>>>
>>>>>>> Use the tools that make sense for the test you're writting.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -Hoss
>>>>>>> http://www.lucidworks.com/
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
>>>>> LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
>>>>> http://www.solrenterprisesearchserver.com
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: What are the expectations for cleanup SolrCloud tests when
tests.iters is specified?
Posted by Robert Muir <rc...@gmail.com>.
just to be clear, as i mentioned: tests.iters should run
setup/teardown just fine, thats not a problem. There is no issue with
tests.iters setup/teardown/Before/After.
but its only going to instantiate your *class* a single time and run
*class*-level stuff (e.g. BeforeClass/AfterClass) a single time.
On Mon, Jan 15, 2018 at 11:35 AM, Erick Erickson
<er...@gmail.com> wrote:
> Any test that doesn't really do anything in setup/teardown seems like a
> good candidate for using tests.iters, which will be much faster.
>
> For SolrCloud level tests beasting is becoming my first choice.
>
> For a lot of the Lucene level code tests.iters makes a lot of sense so
> it's too useful to remove.
>
> Best,
> Erick
>
>
> On Mon, Jan 15, 2018 at 7:56 AM, Jason Gerlowski <ge...@gmail.com> wrote:
>> It sounds like the recommendation in this thread is to _always_ use
>> "ant beast" instead of "tests.iters". Is there _any_ case where
>> "tests.iters" should be preferred? If not, should we remove support
>> for "tests.iters" to remove any ambiguity? (Especially since this has
>> come up on the list a few times...)
>>
>> On Fri, Jan 12, 2018 at 11:14 AM, Erick Erickson
>> <er...@gmail.com> wrote:
>>> Well, since I'm in there anyway I'll include the note in the patch. At
>>> least that'll alert people to dig deeper.
>>>
>>> On Thu, Jan 11, 2018 at 8:34 PM, David Smiley <da...@gmail.com> wrote:
>>>> Yeah thanks guys -- beast it is.
>>>>
>>>> I wonder if we should not document tests.iters (a bit more expert), or add a
>>>> warning to it in the help output saying something like: NOTE: some tests are
>>>> incompatible because BeforeClass/AfterClass isn't performed inbetween. Try
>>>> beast.iters instead.
>>>>
>>>> On Thu, Jan 11, 2018 at 8:39 PM Erick Erickson <er...@gmail.com>
>>>> wrote:
>>>>>
>>>>> Ok, thanks both. That makes a lot of sense. I'll just use beasting for
>>>>> most anything SolrCloud related.
>>>>>
>>>>>
>>>>> On Thu, Jan 11, 2018 at 4:56 PM, Chris Hostetter
>>>>> <ho...@fucit.org> wrote:
>>>>>>
>>>>>> : (I had left the comment in question)
>>>>>> : I think a test shouldn't have to explicitly clean up after itself,
>>>>>> except
>>>>>> : perhaps intra-method as-needed; test-infrastructure should do the class
>>>>>> : (test suite).
>>>>>>
>>>>>> All test code should always be expected to clean up their messes at
>>>>>> whatever "ending" stage corrisponds with the stage where the mess was
>>>>>> made.
>>>>>>
>>>>>> how the mess is cleaned up, and wether infrastructure/scaffolding code
>>>>>> helps do that dpeends on the specifics of the infrastucture/scaffolding
>>>>>> in
>>>>>> question -- if you make a mess in a test method that the general purpose
>>>>>> infrastructure doesn't expect, then the burden is on you
>>>>>> to add the level of infrastructure (either in your specific test class,
>>>>>> or
>>>>>> in a new abstract base class depending on how you think it might be
>>>>>> re-usable) to do so.
>>>>>>
>>>>>> In the abstract: Assume AbstractParentTest class that creates some
>>>>>> "parentMess" in @BeforeClass, and deletes "parentMess" in an
>>>>>> @AfterClass....
>>>>>>
>>>>>> 1) if you want all of your tests methods to have access to a
>>>>>> shiny new/unique instance of "childMess" in every test method, then
>>>>>> burden
>>>>>> is on you to create/destroy childMess in your own @Before and @After
>>>>>> methods
>>>>>>
>>>>>> 2) If you want test methods that are going to mutate "parentMess" then
>>>>>> the
>>>>>> burden is on you to ensure (ideally via @After methods that "do the right
>>>>>> thing" even if the test method fails) that "parentMess" is correctly
>>>>>> reset
>>>>>> so that all the test methods depending on "parentMess" can run in any
>>>>>> order (or run multiple times in a single instance) ... either that, or
>>>>>> you
>>>>>> shouldn't use AbstractParentTest -- you should create/destroy
>>>>>> a "parentMess" instance yourself in your @Before & @After methods
>>>>>>
>>>>>> Concretely...
>>>>>>
>>>>>> : > The assumption was that everything would be cleaned up between runs
>>>>>> : > doesn't appear to be true for SolrCloud tests. I think one of two
>>>>>> things is
>>>>>> : > happening:
>>>>>> : >
>>>>>> : > 1> collections (and perhaps aliases) are simply not cleaned up
>>>>>> : >
>>>>>> : > 2> there is a timing issue, we have waitForCollectionToDisappear in
>>>>>> test
>>>>>> : > code after all.
>>>>>>
>>>>>> ...these are vague statements ("everything", "SolrCloud tests", "not
>>>>>> cleaned up") and not being intimately familiar with the test class in
>>>>>> question it's not clear exactly is happening or what expectations various
>>>>>> people have -- BUT -- assuming this is in regards to
>>>>>> SolrCloudTestCase, that base class has very explicit docs about
>>>>>> how it's intended to be used: you are expected to configure & init a
>>>>>> MiniSolrCloudCluster instance in an @BeforeClass method -- it has helper
>>>>>> code for this -- and that cluster lives for the lifespan of the class at
>>>>>> which point an @AfterClass in SolrCloudTestCase will ensure it gets torn
>>>>>> down.
>>>>>>
>>>>>> Tests which subclass SolrCloudTestCase should be initializing the cluster
>>>>>> only in @BeforeClass. Most tests should only be creating collections in
>>>>>> @BeforeClass -- allthough you are certainly free to do things like
>>>>>> create/destroy collections on a per test method basis in @Before/@After
>>>>>> methods if you have a need for that sort of thing.
>>>>>>
>>>>>> If that's not the lifecycle you want -- if you want a lifecycle where
>>>>>> ever
>>>>>> individual test method gets it's own pristine new MiniSolrCloudCluster
>>>>>> instance w/o any pre-existing collections, then you shouldn't use
>>>>>> SolrCloudTestCase -- you should just create/destroy
>>>>>> unique MiniSolrCloudCluster instances in your own @Before/@After methods.
>>>>>>
>>>>>>
>>>>>> Bottom Line: there is no one size fits all test scaffolding -- not when
>>>>>> we
>>>>>> have some tests classes where we want to create a collection once, fill
>>>>>> it
>>>>>> with lots of docs, and then re-use it in 100s of test methods, but other
>>>>>> classes want to test the very operation of creating/deleting collections.
>>>>>>
>>>>>> Use the tools that make sense for the test you're writting.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -Hoss
>>>>>> http://www.lucidworks.com/
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
>>>> LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
>>>> http://www.solrenterprisesearchserver.com
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: What are the expectations for cleanup SolrCloud tests when
tests.iters is specified?
Posted by Erick Erickson <er...@gmail.com>.
Any test that doesn't really do anything in setup/teardown seems like a
good candidate for using tests.iters, which will be much faster.
For SolrCloud level tests beasting is becoming my first choice.
For a lot of the Lucene level code tests.iters makes a lot of sense so
it's too useful to remove.
Best,
Erick
On Mon, Jan 15, 2018 at 7:56 AM, Jason Gerlowski <ge...@gmail.com> wrote:
> It sounds like the recommendation in this thread is to _always_ use
> "ant beast" instead of "tests.iters". Is there _any_ case where
> "tests.iters" should be preferred? If not, should we remove support
> for "tests.iters" to remove any ambiguity? (Especially since this has
> come up on the list a few times...)
>
> On Fri, Jan 12, 2018 at 11:14 AM, Erick Erickson
> <er...@gmail.com> wrote:
>> Well, since I'm in there anyway I'll include the note in the patch. At
>> least that'll alert people to dig deeper.
>>
>> On Thu, Jan 11, 2018 at 8:34 PM, David Smiley <da...@gmail.com> wrote:
>>> Yeah thanks guys -- beast it is.
>>>
>>> I wonder if we should not document tests.iters (a bit more expert), or add a
>>> warning to it in the help output saying something like: NOTE: some tests are
>>> incompatible because BeforeClass/AfterClass isn't performed inbetween. Try
>>> beast.iters instead.
>>>
>>> On Thu, Jan 11, 2018 at 8:39 PM Erick Erickson <er...@gmail.com>
>>> wrote:
>>>>
>>>> Ok, thanks both. That makes a lot of sense. I'll just use beasting for
>>>> most anything SolrCloud related.
>>>>
>>>>
>>>> On Thu, Jan 11, 2018 at 4:56 PM, Chris Hostetter
>>>> <ho...@fucit.org> wrote:
>>>>>
>>>>> : (I had left the comment in question)
>>>>> : I think a test shouldn't have to explicitly clean up after itself,
>>>>> except
>>>>> : perhaps intra-method as-needed; test-infrastructure should do the class
>>>>> : (test suite).
>>>>>
>>>>> All test code should always be expected to clean up their messes at
>>>>> whatever "ending" stage corrisponds with the stage where the mess was
>>>>> made.
>>>>>
>>>>> how the mess is cleaned up, and wether infrastructure/scaffolding code
>>>>> helps do that dpeends on the specifics of the infrastucture/scaffolding
>>>>> in
>>>>> question -- if you make a mess in a test method that the general purpose
>>>>> infrastructure doesn't expect, then the burden is on you
>>>>> to add the level of infrastructure (either in your specific test class,
>>>>> or
>>>>> in a new abstract base class depending on how you think it might be
>>>>> re-usable) to do so.
>>>>>
>>>>> In the abstract: Assume AbstractParentTest class that creates some
>>>>> "parentMess" in @BeforeClass, and deletes "parentMess" in an
>>>>> @AfterClass....
>>>>>
>>>>> 1) if you want all of your tests methods to have access to a
>>>>> shiny new/unique instance of "childMess" in every test method, then
>>>>> burden
>>>>> is on you to create/destroy childMess in your own @Before and @After
>>>>> methods
>>>>>
>>>>> 2) If you want test methods that are going to mutate "parentMess" then
>>>>> the
>>>>> burden is on you to ensure (ideally via @After methods that "do the right
>>>>> thing" even if the test method fails) that "parentMess" is correctly
>>>>> reset
>>>>> so that all the test methods depending on "parentMess" can run in any
>>>>> order (or run multiple times in a single instance) ... either that, or
>>>>> you
>>>>> shouldn't use AbstractParentTest -- you should create/destroy
>>>>> a "parentMess" instance yourself in your @Before & @After methods
>>>>>
>>>>> Concretely...
>>>>>
>>>>> : > The assumption was that everything would be cleaned up between runs
>>>>> : > doesn't appear to be true for SolrCloud tests. I think one of two
>>>>> things is
>>>>> : > happening:
>>>>> : >
>>>>> : > 1> collections (and perhaps aliases) are simply not cleaned up
>>>>> : >
>>>>> : > 2> there is a timing issue, we have waitForCollectionToDisappear in
>>>>> test
>>>>> : > code after all.
>>>>>
>>>>> ...these are vague statements ("everything", "SolrCloud tests", "not
>>>>> cleaned up") and not being intimately familiar with the test class in
>>>>> question it's not clear exactly is happening or what expectations various
>>>>> people have -- BUT -- assuming this is in regards to
>>>>> SolrCloudTestCase, that base class has very explicit docs about
>>>>> how it's intended to be used: you are expected to configure & init a
>>>>> MiniSolrCloudCluster instance in an @BeforeClass method -- it has helper
>>>>> code for this -- and that cluster lives for the lifespan of the class at
>>>>> which point an @AfterClass in SolrCloudTestCase will ensure it gets torn
>>>>> down.
>>>>>
>>>>> Tests which subclass SolrCloudTestCase should be initializing the cluster
>>>>> only in @BeforeClass. Most tests should only be creating collections in
>>>>> @BeforeClass -- allthough you are certainly free to do things like
>>>>> create/destroy collections on a per test method basis in @Before/@After
>>>>> methods if you have a need for that sort of thing.
>>>>>
>>>>> If that's not the lifecycle you want -- if you want a lifecycle where
>>>>> ever
>>>>> individual test method gets it's own pristine new MiniSolrCloudCluster
>>>>> instance w/o any pre-existing collections, then you shouldn't use
>>>>> SolrCloudTestCase -- you should just create/destroy
>>>>> unique MiniSolrCloudCluster instances in your own @Before/@After methods.
>>>>>
>>>>>
>>>>> Bottom Line: there is no one size fits all test scaffolding -- not when
>>>>> we
>>>>> have some tests classes where we want to create a collection once, fill
>>>>> it
>>>>> with lots of docs, and then re-use it in 100s of test methods, but other
>>>>> classes want to test the very operation of creating/deleting collections.
>>>>>
>>>>> Use the tools that make sense for the test you're writting.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -Hoss
>>>>> http://www.lucidworks.com/
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>>
>>>>
>>>
>>>
>>> --
>>> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
>>> LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
>>> http://www.solrenterprisesearchserver.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: What are the expectations for cleanup SolrCloud tests when
tests.iters is specified?
Posted by Jason Gerlowski <ge...@gmail.com>.
It sounds like the recommendation in this thread is to _always_ use
"ant beast" instead of "tests.iters". Is there _any_ case where
"tests.iters" should be preferred? If not, should we remove support
for "tests.iters" to remove any ambiguity? (Especially since this has
come up on the list a few times...)
On Fri, Jan 12, 2018 at 11:14 AM, Erick Erickson
<er...@gmail.com> wrote:
> Well, since I'm in there anyway I'll include the note in the patch. At
> least that'll alert people to dig deeper.
>
> On Thu, Jan 11, 2018 at 8:34 PM, David Smiley <da...@gmail.com> wrote:
>> Yeah thanks guys -- beast it is.
>>
>> I wonder if we should not document tests.iters (a bit more expert), or add a
>> warning to it in the help output saying something like: NOTE: some tests are
>> incompatible because BeforeClass/AfterClass isn't performed inbetween. Try
>> beast.iters instead.
>>
>> On Thu, Jan 11, 2018 at 8:39 PM Erick Erickson <er...@gmail.com>
>> wrote:
>>>
>>> Ok, thanks both. That makes a lot of sense. I'll just use beasting for
>>> most anything SolrCloud related.
>>>
>>>
>>> On Thu, Jan 11, 2018 at 4:56 PM, Chris Hostetter
>>> <ho...@fucit.org> wrote:
>>>>
>>>> : (I had left the comment in question)
>>>> : I think a test shouldn't have to explicitly clean up after itself,
>>>> except
>>>> : perhaps intra-method as-needed; test-infrastructure should do the class
>>>> : (test suite).
>>>>
>>>> All test code should always be expected to clean up their messes at
>>>> whatever "ending" stage corrisponds with the stage where the mess was
>>>> made.
>>>>
>>>> how the mess is cleaned up, and wether infrastructure/scaffolding code
>>>> helps do that dpeends on the specifics of the infrastucture/scaffolding
>>>> in
>>>> question -- if you make a mess in a test method that the general purpose
>>>> infrastructure doesn't expect, then the burden is on you
>>>> to add the level of infrastructure (either in your specific test class,
>>>> or
>>>> in a new abstract base class depending on how you think it might be
>>>> re-usable) to do so.
>>>>
>>>> In the abstract: Assume AbstractParentTest class that creates some
>>>> "parentMess" in @BeforeClass, and deletes "parentMess" in an
>>>> @AfterClass....
>>>>
>>>> 1) if you want all of your tests methods to have access to a
>>>> shiny new/unique instance of "childMess" in every test method, then
>>>> burden
>>>> is on you to create/destroy childMess in your own @Before and @After
>>>> methods
>>>>
>>>> 2) If you want test methods that are going to mutate "parentMess" then
>>>> the
>>>> burden is on you to ensure (ideally via @After methods that "do the right
>>>> thing" even if the test method fails) that "parentMess" is correctly
>>>> reset
>>>> so that all the test methods depending on "parentMess" can run in any
>>>> order (or run multiple times in a single instance) ... either that, or
>>>> you
>>>> shouldn't use AbstractParentTest -- you should create/destroy
>>>> a "parentMess" instance yourself in your @Before & @After methods
>>>>
>>>> Concretely...
>>>>
>>>> : > The assumption was that everything would be cleaned up between runs
>>>> : > doesn't appear to be true for SolrCloud tests. I think one of two
>>>> things is
>>>> : > happening:
>>>> : >
>>>> : > 1> collections (and perhaps aliases) are simply not cleaned up
>>>> : >
>>>> : > 2> there is a timing issue, we have waitForCollectionToDisappear in
>>>> test
>>>> : > code after all.
>>>>
>>>> ...these are vague statements ("everything", "SolrCloud tests", "not
>>>> cleaned up") and not being intimately familiar with the test class in
>>>> question it's not clear exactly is happening or what expectations various
>>>> people have -- BUT -- assuming this is in regards to
>>>> SolrCloudTestCase, that base class has very explicit docs about
>>>> how it's intended to be used: you are expected to configure & init a
>>>> MiniSolrCloudCluster instance in an @BeforeClass method -- it has helper
>>>> code for this -- and that cluster lives for the lifespan of the class at
>>>> which point an @AfterClass in SolrCloudTestCase will ensure it gets torn
>>>> down.
>>>>
>>>> Tests which subclass SolrCloudTestCase should be initializing the cluster
>>>> only in @BeforeClass. Most tests should only be creating collections in
>>>> @BeforeClass -- allthough you are certainly free to do things like
>>>> create/destroy collections on a per test method basis in @Before/@After
>>>> methods if you have a need for that sort of thing.
>>>>
>>>> If that's not the lifecycle you want -- if you want a lifecycle where
>>>> ever
>>>> individual test method gets it's own pristine new MiniSolrCloudCluster
>>>> instance w/o any pre-existing collections, then you shouldn't use
>>>> SolrCloudTestCase -- you should just create/destroy
>>>> unique MiniSolrCloudCluster instances in your own @Before/@After methods.
>>>>
>>>>
>>>> Bottom Line: there is no one size fits all test scaffolding -- not when
>>>> we
>>>> have some tests classes where we want to create a collection once, fill
>>>> it
>>>> with lots of docs, and then re-use it in 100s of test methods, but other
>>>> classes want to test the very operation of creating/deleting collections.
>>>>
>>>> Use the tools that make sense for the test you're writting.
>>>>
>>>>
>>>>
>>>>
>>>> -Hoss
>>>> http://www.lucidworks.com/
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>>
>>>
>>
>>
>> --
>> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
>> LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
>> http://www.solrenterprisesearchserver.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: What are the expectations for cleanup SolrCloud tests when
tests.iters is specified?
Posted by Erick Erickson <er...@gmail.com>.
Well, since I'm in there anyway I'll include the note in the patch. At
least that'll alert people to dig deeper.
On Thu, Jan 11, 2018 at 8:34 PM, David Smiley <da...@gmail.com> wrote:
> Yeah thanks guys -- beast it is.
>
> I wonder if we should not document tests.iters (a bit more expert), or add a
> warning to it in the help output saying something like: NOTE: some tests are
> incompatible because BeforeClass/AfterClass isn't performed inbetween. Try
> beast.iters instead.
>
> On Thu, Jan 11, 2018 at 8:39 PM Erick Erickson <er...@gmail.com>
> wrote:
>>
>> Ok, thanks both. That makes a lot of sense. I'll just use beasting for
>> most anything SolrCloud related.
>>
>>
>> On Thu, Jan 11, 2018 at 4:56 PM, Chris Hostetter
>> <ho...@fucit.org> wrote:
>>>
>>> : (I had left the comment in question)
>>> : I think a test shouldn't have to explicitly clean up after itself,
>>> except
>>> : perhaps intra-method as-needed; test-infrastructure should do the class
>>> : (test suite).
>>>
>>> All test code should always be expected to clean up their messes at
>>> whatever "ending" stage corrisponds with the stage where the mess was
>>> made.
>>>
>>> how the mess is cleaned up, and wether infrastructure/scaffolding code
>>> helps do that dpeends on the specifics of the infrastucture/scaffolding
>>> in
>>> question -- if you make a mess in a test method that the general purpose
>>> infrastructure doesn't expect, then the burden is on you
>>> to add the level of infrastructure (either in your specific test class,
>>> or
>>> in a new abstract base class depending on how you think it might be
>>> re-usable) to do so.
>>>
>>> In the abstract: Assume AbstractParentTest class that creates some
>>> "parentMess" in @BeforeClass, and deletes "parentMess" in an
>>> @AfterClass....
>>>
>>> 1) if you want all of your tests methods to have access to a
>>> shiny new/unique instance of "childMess" in every test method, then
>>> burden
>>> is on you to create/destroy childMess in your own @Before and @After
>>> methods
>>>
>>> 2) If you want test methods that are going to mutate "parentMess" then
>>> the
>>> burden is on you to ensure (ideally via @After methods that "do the right
>>> thing" even if the test method fails) that "parentMess" is correctly
>>> reset
>>> so that all the test methods depending on "parentMess" can run in any
>>> order (or run multiple times in a single instance) ... either that, or
>>> you
>>> shouldn't use AbstractParentTest -- you should create/destroy
>>> a "parentMess" instance yourself in your @Before & @After methods
>>>
>>> Concretely...
>>>
>>> : > The assumption was that everything would be cleaned up between runs
>>> : > doesn't appear to be true for SolrCloud tests. I think one of two
>>> things is
>>> : > happening:
>>> : >
>>> : > 1> collections (and perhaps aliases) are simply not cleaned up
>>> : >
>>> : > 2> there is a timing issue, we have waitForCollectionToDisappear in
>>> test
>>> : > code after all.
>>>
>>> ...these are vague statements ("everything", "SolrCloud tests", "not
>>> cleaned up") and not being intimately familiar with the test class in
>>> question it's not clear exactly is happening or what expectations various
>>> people have -- BUT -- assuming this is in regards to
>>> SolrCloudTestCase, that base class has very explicit docs about
>>> how it's intended to be used: you are expected to configure & init a
>>> MiniSolrCloudCluster instance in an @BeforeClass method -- it has helper
>>> code for this -- and that cluster lives for the lifespan of the class at
>>> which point an @AfterClass in SolrCloudTestCase will ensure it gets torn
>>> down.
>>>
>>> Tests which subclass SolrCloudTestCase should be initializing the cluster
>>> only in @BeforeClass. Most tests should only be creating collections in
>>> @BeforeClass -- allthough you are certainly free to do things like
>>> create/destroy collections on a per test method basis in @Before/@After
>>> methods if you have a need for that sort of thing.
>>>
>>> If that's not the lifecycle you want -- if you want a lifecycle where
>>> ever
>>> individual test method gets it's own pristine new MiniSolrCloudCluster
>>> instance w/o any pre-existing collections, then you shouldn't use
>>> SolrCloudTestCase -- you should just create/destroy
>>> unique MiniSolrCloudCluster instances in your own @Before/@After methods.
>>>
>>>
>>> Bottom Line: there is no one size fits all test scaffolding -- not when
>>> we
>>> have some tests classes where we want to create a collection once, fill
>>> it
>>> with lots of docs, and then re-use it in 100s of test methods, but other
>>> classes want to test the very operation of creating/deleting collections.
>>>
>>> Use the tools that make sense for the test you're writting.
>>>
>>>
>>>
>>>
>>> -Hoss
>>> http://www.lucidworks.com/
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>
>
> --
> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
> LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
> http://www.solrenterprisesearchserver.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: What are the expectations for cleanup SolrCloud tests when
tests.iters is specified?
Posted by David Smiley <da...@gmail.com>.
Yeah thanks guys -- beast it is.
I wonder if we should not document tests.iters (a bit more expert), or add
a warning to it in the help output saying something like: NOTE: some tests
are incompatible because BeforeClass/AfterClass isn't performed inbetween.
Try beast.iters instead.
On Thu, Jan 11, 2018 at 8:39 PM Erick Erickson <er...@gmail.com>
wrote:
> Ok, thanks both. That makes a lot of sense. I'll just use beasting for
> most anything SolrCloud related.
>
>
> On Thu, Jan 11, 2018 at 4:56 PM, Chris Hostetter <hossman_lucene@fucit.org
> > wrote:
>
>> : (I had left the comment in question)
>> : I think a test shouldn't have to explicitly clean up after itself,
>> except
>> : perhaps intra-method as-needed; test-infrastructure should do the class
>> : (test suite).
>>
>> All test code should always be expected to clean up their messes at
>> whatever "ending" stage corrisponds with the stage where the mess was
>> made.
>>
>> how the mess is cleaned up, and wether infrastructure/scaffolding code
>> helps do that dpeends on the specifics of the infrastucture/scaffolding in
>> question -- if you make a mess in a test method that the general purpose
>> infrastructure doesn't expect, then the burden is on you
>> to add the level of infrastructure (either in your specific test class, or
>> in a new abstract base class depending on how you think it might be
>> re-usable) to do so.
>>
>> In the abstract: Assume AbstractParentTest class that creates some
>> "parentMess" in @BeforeClass, and deletes "parentMess" in an
>> @AfterClass....
>>
>> 1) if you want all of your tests methods to have access to a
>> shiny new/unique instance of "childMess" in every test method, then burden
>> is on you to create/destroy childMess in your own @Before and @After
>> methods
>>
>> 2) If you want test methods that are going to mutate "parentMess" then the
>> burden is on you to ensure (ideally via @After methods that "do the right
>> thing" even if the test method fails) that "parentMess" is correctly reset
>> so that all the test methods depending on "parentMess" can run in any
>> order (or run multiple times in a single instance) ... either that, or you
>> shouldn't use AbstractParentTest -- you should create/destroy
>> a "parentMess" instance yourself in your @Before & @After methods
>>
>> Concretely...
>>
>> : > The assumption was that everything would be cleaned up between runs
>> : > doesn't appear to be true for SolrCloud tests. I think one of two
>> things is
>> : > happening:
>> : >
>> : > 1> collections (and perhaps aliases) are simply not cleaned up
>> : >
>> : > 2> there is a timing issue, we have waitForCollectionToDisappear in
>> test
>> : > code after all.
>>
>> ...these are vague statements ("everything", "SolrCloud tests", "not
>> cleaned up") and not being intimately familiar with the test class in
>> question it's not clear exactly is happening or what expectations various
>> people have -- BUT -- assuming this is in regards to
>> SolrCloudTestCase, that base class has very explicit docs about
>> how it's intended to be used: you are expected to configure & init a
>> MiniSolrCloudCluster instance in an @BeforeClass method -- it has helper
>> code for this -- and that cluster lives for the lifespan of the class at
>> which point an @AfterClass in SolrCloudTestCase will ensure it gets torn
>> down.
>>
>> Tests which subclass SolrCloudTestCase should be initializing the cluster
>> only in @BeforeClass. Most tests should only be creating collections in
>> @BeforeClass -- allthough you are certainly free to do things like
>> create/destroy collections on a per test method basis in @Before/@After
>> methods if you have a need for that sort of thing.
>>
>> If that's not the lifecycle you want -- if you want a lifecycle where ever
>> individual test method gets it's own pristine new MiniSolrCloudCluster
>> instance w/o any pre-existing collections, then you shouldn't use
>> SolrCloudTestCase -- you should just create/destroy
>> unique MiniSolrCloudCluster instances in your own @Before/@After methods.
>>
>>
>> Bottom Line: there is no one size fits all test scaffolding -- not when we
>> have some tests classes where we want to create a collection once, fill it
>> with lots of docs, and then re-use it in 100s of test methods, but other
>> classes want to test the very operation of creating/deleting collections.
>>
>> Use the tools that make sense for the test you're writting.
>>
>>
>>
>>
>> -Hoss
>> http://www.lucidworks.com/
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>
--
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com
Re: What are the expectations for cleanup SolrCloud tests when
tests.iters is specified?
Posted by Erick Erickson <er...@gmail.com>.
Ok, thanks both. That makes a lot of sense. I'll just use beasting for
most anything SolrCloud related.
On Thu, Jan 11, 2018 at 4:56 PM, Chris Hostetter <ho...@fucit.org>
wrote:
> : (I had left the comment in question)
> : I think a test shouldn't have to explicitly clean up after itself, except
> : perhaps intra-method as-needed; test-infrastructure should do the class
> : (test suite).
>
> All test code should always be expected to clean up their messes at
> whatever "ending" stage corrisponds with the stage where the mess was
> made.
>
> how the mess is cleaned up, and wether infrastructure/scaffolding code
> helps do that dpeends on the specifics of the infrastucture/scaffolding in
> question -- if you make a mess in a test method that the general purpose
> infrastructure doesn't expect, then the burden is on you
> to add the level of infrastructure (either in your specific test class, or
> in a new abstract base class depending on how you think it might be
> re-usable) to do so.
>
> In the abstract: Assume AbstractParentTest class that creates some
> "parentMess" in @BeforeClass, and deletes "parentMess" in an
> @AfterClass....
>
> 1) if you want all of your tests methods to have access to a
> shiny new/unique instance of "childMess" in every test method, then burden
> is on you to create/destroy childMess in your own @Before and @After
> methods
>
> 2) If you want test methods that are going to mutate "parentMess" then the
> burden is on you to ensure (ideally via @After methods that "do the right
> thing" even if the test method fails) that "parentMess" is correctly reset
> so that all the test methods depending on "parentMess" can run in any
> order (or run multiple times in a single instance) ... either that, or you
> shouldn't use AbstractParentTest -- you should create/destroy
> a "parentMess" instance yourself in your @Before & @After methods
>
> Concretely...
>
> : > The assumption was that everything would be cleaned up between runs
> : > doesn't appear to be true for SolrCloud tests. I think one of two
> things is
> : > happening:
> : >
> : > 1> collections (and perhaps aliases) are simply not cleaned up
> : >
> : > 2> there is a timing issue, we have waitForCollectionToDisappear in
> test
> : > code after all.
>
> ...these are vague statements ("everything", "SolrCloud tests", "not
> cleaned up") and not being intimately familiar with the test class in
> question it's not clear exactly is happening or what expectations various
> people have -- BUT -- assuming this is in regards to
> SolrCloudTestCase, that base class has very explicit docs about
> how it's intended to be used: you are expected to configure & init a
> MiniSolrCloudCluster instance in an @BeforeClass method -- it has helper
> code for this -- and that cluster lives for the lifespan of the class at
> which point an @AfterClass in SolrCloudTestCase will ensure it gets torn
> down.
>
> Tests which subclass SolrCloudTestCase should be initializing the cluster
> only in @BeforeClass. Most tests should only be creating collections in
> @BeforeClass -- allthough you are certainly free to do things like
> create/destroy collections on a per test method basis in @Before/@After
> methods if you have a need for that sort of thing.
>
> If that's not the lifecycle you want -- if you want a lifecycle where ever
> individual test method gets it's own pristine new MiniSolrCloudCluster
> instance w/o any pre-existing collections, then you shouldn't use
> SolrCloudTestCase -- you should just create/destroy
> unique MiniSolrCloudCluster instances in your own @Before/@After methods.
>
>
> Bottom Line: there is no one size fits all test scaffolding -- not when we
> have some tests classes where we want to create a collection once, fill it
> with lots of docs, and then re-use it in 100s of test methods, but other
> classes want to test the very operation of creating/deleting collections.
>
> Use the tools that make sense for the test you're writting.
>
>
>
>
> -Hoss
> http://www.lucidworks.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
Re: What are the expectations for cleanup SolrCloud tests when
tests.iters is specified?
Posted by Chris Hostetter <ho...@fucit.org>.
: (I had left the comment in question)
: I think a test shouldn't have to explicitly clean up after itself, except
: perhaps intra-method as-needed; test-infrastructure should do the class
: (test suite).
All test code should always be expected to clean up their messes at
whatever "ending" stage corrisponds with the stage where the mess was
made.
how the mess is cleaned up, and wether infrastructure/scaffolding code
helps do that dpeends on the specifics of the infrastucture/scaffolding in
question -- if you make a mess in a test method that the general purpose
infrastructure doesn't expect, then the burden is on you
to add the level of infrastructure (either in your specific test class, or
in a new abstract base class depending on how you think it might be
re-usable) to do so.
In the abstract: Assume AbstractParentTest class that creates some
"parentMess" in @BeforeClass, and deletes "parentMess" in an
@AfterClass....
1) if you want all of your tests methods to have access to a
shiny new/unique instance of "childMess" in every test method, then burden
is on you to create/destroy childMess in your own @Before and @After
methods
2) If you want test methods that are going to mutate "parentMess" then the
burden is on you to ensure (ideally via @After methods that "do the right
thing" even if the test method fails) that "parentMess" is correctly reset
so that all the test methods depending on "parentMess" can run in any
order (or run multiple times in a single instance) ... either that, or you
shouldn't use AbstractParentTest -- you should create/destroy
a "parentMess" instance yourself in your @Before & @After methods
Concretely...
: > The assumption was that everything would be cleaned up between runs
: > doesn't appear to be true for SolrCloud tests. I think one of two things is
: > happening:
: >
: > 1> collections (and perhaps aliases) are simply not cleaned up
: >
: > 2> there is a timing issue, we have waitForCollectionToDisappear in test
: > code after all.
...these are vague statements ("everything", "SolrCloud tests", "not
cleaned up") and not being intimately familiar with the test class in
question it's not clear exactly is happening or what expectations various
people have -- BUT -- assuming this is in regards to
SolrCloudTestCase, that base class has very explicit docs about
how it's intended to be used: you are expected to configure & init a
MiniSolrCloudCluster instance in an @BeforeClass method -- it has helper
code for this -- and that cluster lives for the lifespan of the class at
which point an @AfterClass in SolrCloudTestCase will ensure it gets torn
down.
Tests which subclass SolrCloudTestCase should be initializing the cluster
only in @BeforeClass. Most tests should only be creating collections in
@BeforeClass -- allthough you are certainly free to do things like
create/destroy collections on a per test method basis in @Before/@After
methods if you have a need for that sort of thing.
If that's not the lifecycle you want -- if you want a lifecycle where ever
individual test method gets it's own pristine new MiniSolrCloudCluster
instance w/o any pre-existing collections, then you shouldn't use
SolrCloudTestCase -- you should just create/destroy
unique MiniSolrCloudCluster instances in your own @Before/@After methods.
Bottom Line: there is no one size fits all test scaffolding -- not when we
have some tests classes where we want to create a collection once, fill it
with lots of docs, and then re-use it in 100s of test methods, but other
classes want to test the very operation of creating/deleting collections.
Use the tools that make sense for the test you're writting.
-Hoss
http://www.lucidworks.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: What are the expectations for cleanup SolrCloud tests when
tests.iters is specified?
Posted by David Smiley <da...@gmail.com>.
(I had left the comment in question)
I think a test shouldn't have to explicitly clean up after itself, except
perhaps intra-method as-needed; test-infrastructure should do the class
(test suite).
On Thu, Jan 11, 2018 at 12:49 PM Erick Erickson <er...@gmail.com>
wrote:
> The recent failures in TimeRoutedAliasUpdateProcessorTest. are a result of
> the changes to SOLR-11218. The initial fix is easy, but while fixing it I
> saw a comment in there about cleaning up when tests.iters is specified > 1.
>
> The assumption was that everything would be cleaned up between runs
> doesn't appear to be true for SolrCloud tests. I think one of two things is
> happening:
>
> 1> collections (and perhaps aliases) are simply not cleaned up
>
> 2> there is a timing issue, we have waitForCollectionToDisappear in test
> code after all.
>
> My question is what are the expectations? It's pretty ugly to require each
> test to clean up all changes to ZooKeeper or else fail when test.iters is
> set. Plus it violates the expectation that running one test doesn't affect
> the other test, even if it's the same test over and over.
>
> So if we think it's an issue with the framework, any clues where to look
> would be appreciated.
>
--
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com
Re: What are the expectations for cleanup SolrCloud tests when
tests.iters is specified?
Posted by Robert Muir <rc...@gmail.com>.
Short answer: don't use tests.iters. Use ant beast.
Long answer: It has been discussed on the list several times.
tests.iters does not do what a lot of people seem to think.
From what i remember: it creates your test *class* a single time and
runs stuff repeatedly... I think only (new *instance*, setup,
teardown, etc). I don't even think it runs e.g. beforeclass/afterclass
stuff between iterations either, but that still wouldnt make it
completely safe anyway unless the whole class was recreated (a test
could leave a static variable in a different state or something).
So if your test isn't a pure unit test, its probably not ready for
tests.iters. Just use ant beast, it works :)
On Thu, Jan 11, 2018 at 12:48 PM, Erick Erickson
<er...@gmail.com> wrote:
> The recent failures in TimeRoutedAliasUpdateProcessorTest. are a result of
> the changes to SOLR-11218. The initial fix is easy, but while fixing it I
> saw a comment in there about cleaning up when tests.iters is specified > 1.
>
> The assumption was that everything would be cleaned up between runs doesn't
> appear to be true for SolrCloud tests. I think one of two things is
> happening:
>
> 1> collections (and perhaps aliases) are simply not cleaned up
>
> 2> there is a timing issue, we have waitForCollectionToDisappear in test
> code after all.
>
> My question is what are the expectations? It's pretty ugly to require each
> test to clean up all changes to ZooKeeper or else fail when test.iters is
> set. Plus it violates the expectation that running one test doesn't affect
> the other test, even if it's the same test over and over.
>
> So if we think it's an issue with the framework, any clues where to look
> would be appreciated.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org