You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Mark Miller <ma...@gmail.com> on 2018/10/25 06:05:39 UTC
Solr Tests
My boss has indicated I'm going to get a little time to spend on addressing
our test situation. Turns out the current downward trend situation is a
little worrying to some that count on the project down the road ;)
We can finally dig out though, I promise.
It's going to be a little bit before I'm clear and gathering steam, but
I've got this figured out. I'm finally playing checkers instead of rock /
paper / scissors.
I'm going to eat a lot of work to get there and I promise I can get there,
but I'm going to need some help.
If you have an interest in changing our test situation, and I know some of
you do, please join me as I invite others to help out more in a short time.
Beyond some help with code effort though, please get involved in
discussions and JIRA issues around changing our test culture and process.
We need to turn this corner as a group, we need to come together on some
new behaviors, we need to generate large enough buy in to do something
lasting.
You all need to learn 'ant beast' (I'm fixing it to work *much* better) and
my beasting test script gist (each has there place). I'm going to force it
on you even if you don't ;)
https://gist.githubusercontent.com/markrmiller/dbdb792216dc98b018ad/raw/cd084d01f405fa271af4b72a2848dc87374a1f71/gistfile1.sh
For a long time I put most of my attention towards SolrCloud stability and
scale. From 2012-2015 or so, that felt like the biggest threat to the
project while also getting relatively little attention. I felt like it was
50/50 in 2014 that I'd even be talking to anyone about SolrCloud today.
That hurdle is done and over with. The biggest threat to Solr and SolrCloud
now is our tests.
I'll be working from this umbrella issue:
https://issues.apache.org/jira/browse/SOLR-12801 : Fix the tests, remove
BadApples and AwaitsFix annotations, improve env for test development.
Join the discussion.
- Mark
--
- Mark
about.me/markrmiller
Re: Solr Tests
Posted by Mark Miller <ma...@gmail.com>.
In response to Erick's question in another thread of: "So, what I can I do
to help?"
I actually have a lot of thoughts on that, but I think I need a little time
to get some things I have in the pipeline fully fleshed out and ready to
share.
At a very high level though...
My thoughts on Solr tests are this:
You can point me to pretty much any failing test and I can fix it. Probably
I'll find 10 things to improve when I do it.
That doesn't seem to be a widespread skill and where it does exist , it is
often under utilized.
So my dilemma is this: I can fix the tests. One by one, over weeks and
months, I can fix the tests. But I cannot stop you from breaking them.
Sometimes I can't stop myself.
It follows that I need a way to stop you and me from breaking the tests
once they are fixed.
And I also need to spread the skill of actually fixing a test in
combination with making it easier so that that skill is learned and not
under utilized. Same as the skill of writing a good test - which is
currently too hard, no doubt.
I have some major ideas on these problems, some of them are in the works.
So at a broad level, you can help by discussing and helping in any of the
JIRA issues I've opened around SOLR-12801
<https://issues.apache.org/jira/browse/SOLR-12801>. Or jump on a test
problem you know of.
Beyond that, once I've started actually fixing and defending tests, you can
help by fixing tests and driving a culture of passing tests. Commits that
change this should be reverted.
I plan to fix and defend tests by package.
In a little time, I'll have much more specific requests.
- Mark
--
- Mark
about.me/markrmiller
Re: Solr Tests
Posted by Mark Miller <ma...@gmail.com>.
Yeah, there has been a lot of chipping for a very long time :)
Many valiant attempts have been made to get ahold of this problem. I've
been part of some of them. It's like zombie whack-a-mole.
That is why you could almost read my email as, oh, again the tests, sure,
that sounds exciting, no has ever tried to fix the tests before, see you
next year.
But this time is different. If I get even a quarter the amount of promised
help I've already heard about, we will get to solid tests this time, and we
will add support to enforce we stay there.
I'm already further along the path than it looks, but that will become more
clear soon.
- Mark
On Thu, Oct 25, 2018 at 11:55 AM Cassandra Targett <ca...@gmail.com>
wrote:
> Hopefully they don't mind too much that I'm speaking for them, but on
> behalf of myself and the 7 other committers that make up the Solr Team at
> Lucidworks, we are also in and eager to help.
>
> We've been trying to chip away at the problem for the past year+, but we
> need a systemic change to really make it better. I will do what I can to
> delay or defer some of our Lucidworks planned tasks to free up as much time
> for people as possible.
>
> On Thu, Oct 25, 2018 at 1:35 AM Anshum Gupta <an...@apple.com> wrote:
>
>> +1 Mark! I’m in.
>>
>> * *Anshum
>>
>>
>> On Oct 24, 2018, at 11:05 PM, Mark Miller <ma...@gmail.com> wrote:
>>
>> My boss has indicated I'm going to get a little time to spend on
>> addressing our test situation. Turns out the current downward trend
>> situation is a little worrying to some that count on the project down the
>> road ;)
>>
>> We can finally dig out though, I promise.
>>
>> It's going to be a little bit before I'm clear and gathering steam, but
>> I've got this figured out. I'm finally playing checkers instead of rock /
>> paper / scissors.
>>
>> I'm going to eat a lot of work to get there and I promise I can get
>> there, but I'm going to need some help.
>>
>> If you have an interest in changing our test situation, and I know some
>> of you do, please join me as I invite others to help out more in a short
>> time.
>>
>> Beyond some help with code effort though, please get involved in
>> discussions and JIRA issues around changing our test culture and process.
>> We need to turn this corner as a group, we need to come together on some
>> new behaviors, we need to generate large enough buy in to do something
>> lasting.
>>
>> You all need to learn 'ant beast' (I'm fixing it to work *much* better)
>> and my beasting test script gist (each has there place). I'm going to force
>> it on you even if you don't ;)
>>
>>
>> https://gist.githubusercontent.com/markrmiller/dbdb792216dc98b018ad/raw/cd084d01f405fa271af4b72a2848dc87374a1f71/gistfile1.sh
>>
>> For a long time I put most of my attention towards SolrCloud stability
>> and scale. From 2012-2015 or so, that felt like the biggest threat to the
>> project while also getting relatively little attention. I felt like it was
>> 50/50 in 2014 that I'd even be talking to anyone about SolrCloud today.
>> That hurdle is done and over with. The biggest threat to Solr and SolrCloud
>> now is our tests.
>>
>> I'll be working from this umbrella issue:
>>
>> https://issues.apache.org/jira/browse/SOLR-12801 : Fix the tests, remove
>> BadApples and AwaitsFix annotations, improve env for test development.
>>
>> Join the discussion.
>>
>> - Mark
>> --
>> - Mark
>> about.me/markrmiller
>>
>>
>> --
- Mark
about.me/markrmiller
Re: Solr Tests
Posted by Cassandra Targett <ca...@gmail.com>.
Hopefully they don't mind too much that I'm speaking for them, but on
behalf of myself and the 7 other committers that make up the Solr Team at
Lucidworks, we are also in and eager to help.
We've been trying to chip away at the problem for the past year+, but we
need a systemic change to really make it better. I will do what I can to
delay or defer some of our Lucidworks planned tasks to free up as much time
for people as possible.
On Thu, Oct 25, 2018 at 1:35 AM Anshum Gupta <an...@apple.com> wrote:
> +1 Mark! I’m in.
>
> * *Anshum
>
>
> On Oct 24, 2018, at 11:05 PM, Mark Miller <ma...@gmail.com> wrote:
>
> My boss has indicated I'm going to get a little time to spend on
> addressing our test situation. Turns out the current downward trend
> situation is a little worrying to some that count on the project down the
> road ;)
>
> We can finally dig out though, I promise.
>
> It's going to be a little bit before I'm clear and gathering steam, but
> I've got this figured out. I'm finally playing checkers instead of rock /
> paper / scissors.
>
> I'm going to eat a lot of work to get there and I promise I can get there,
> but I'm going to need some help.
>
> If you have an interest in changing our test situation, and I know some of
> you do, please join me as I invite others to help out more in a short time.
>
> Beyond some help with code effort though, please get involved in
> discussions and JIRA issues around changing our test culture and process.
> We need to turn this corner as a group, we need to come together on some
> new behaviors, we need to generate large enough buy in to do something
> lasting.
>
> You all need to learn 'ant beast' (I'm fixing it to work *much* better)
> and my beasting test script gist (each has there place). I'm going to force
> it on you even if you don't ;)
>
>
> https://gist.githubusercontent.com/markrmiller/dbdb792216dc98b018ad/raw/cd084d01f405fa271af4b72a2848dc87374a1f71/gistfile1.sh
>
> For a long time I put most of my attention towards SolrCloud stability and
> scale. From 2012-2015 or so, that felt like the biggest threat to the
> project while also getting relatively little attention. I felt like it was
> 50/50 in 2014 that I'd even be talking to anyone about SolrCloud today.
> That hurdle is done and over with. The biggest threat to Solr and SolrCloud
> now is our tests.
>
> I'll be working from this umbrella issue:
>
> https://issues.apache.org/jira/browse/SOLR-12801 : Fix the tests, remove
> BadApples and AwaitsFix annotations, improve env for test development.
>
> Join the discussion.
>
> - Mark
> --
> - Mark
> about.me/markrmiller
>
>
>
Re: Solr Tests
Posted by Erick Erickson <er...@gmail.com>.
Well, winter is coming on with long cold nights. What better thing to
do than hunch over my computer working with tests?
+1, I'll certainly add what I can
On Wed, Oct 24, 2018 at 11:35 PM Anshum Gupta <an...@apple.com> wrote:
>
> +1 Mark! I’m in.
>
> Anshum
>
>
> On Oct 24, 2018, at 11:05 PM, Mark Miller <ma...@gmail.com> wrote:
>
> My boss has indicated I'm going to get a little time to spend on addressing our test situation. Turns out the current downward trend situation is a little worrying to some that count on the project down the road ;)
>
> We can finally dig out though, I promise.
>
> It's going to be a little bit before I'm clear and gathering steam, but I've got this figured out. I'm finally playing checkers instead of rock / paper / scissors.
>
> I'm going to eat a lot of work to get there and I promise I can get there, but I'm going to need some help.
>
> If you have an interest in changing our test situation, and I know some of you do, please join me as I invite others to help out more in a short time.
>
> Beyond some help with code effort though, please get involved in discussions and JIRA issues around changing our test culture and process. We need to turn this corner as a group, we need to come together on some new behaviors, we need to generate large enough buy in to do something lasting.
>
> You all need to learn 'ant beast' (I'm fixing it to work *much* better) and my beasting test script gist (each has there place). I'm going to force it on you even if you don't ;)
>
> https://gist.githubusercontent.com/markrmiller/dbdb792216dc98b018ad/raw/cd084d01f405fa271af4b72a2848dc87374a1f71/gistfile1.sh
>
> For a long time I put most of my attention towards SolrCloud stability and scale. From 2012-2015 or so, that felt like the biggest threat to the project while also getting relatively little attention. I felt like it was 50/50 in 2014 that I'd even be talking to anyone about SolrCloud today. That hurdle is done and over with. The biggest threat to Solr and SolrCloud now is our tests.
>
> I'll be working from this umbrella issue:
>
> https://issues.apache.org/jira/browse/SOLR-12801 : Fix the tests, remove BadApples and AwaitsFix annotations, improve env for test development.
>
> Join the discussion.
>
> - Mark
> --
> - Mark
> about.me/markrmiller
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Solr Tests
Posted by Gus Heck <gu...@gmail.com>.
This situation has irritated me to no end and "something to help tests" is
on my short list of things to contribute, along with "streaming expression
stuff", "what ever happened to 3191" and of course continuing to help with
TRA's
Count me in.
On Thu, Oct 25, 2018, 2:35 AM Anshum Gupta <anshumg@apple.com wrote:
> +1 Mark! I’m in.
>
> * *Anshum
>
>
> On Oct 24, 2018, at 11:05 PM, Mark Miller <ma...@gmail.com> wrote:
>
> My boss has indicated I'm going to get a little time to spend on
> addressing our test situation. Turns out the current downward trend
> situation is a little worrying to some that count on the project down the
> road ;)
>
> We can finally dig out though, I promise.
>
> It's going to be a little bit before I'm clear and gathering steam, but
> I've got this figured out. I'm finally playing checkers instead of rock /
> paper / scissors.
>
> I'm going to eat a lot of work to get there and I promise I can get there,
> but I'm going to need some help.
>
> If you have an interest in changing our test situation, and I know some of
> you do, please join me as I invite others to help out more in a short time.
>
> Beyond some help with code effort though, please get involved in
> discussions and JIRA issues around changing our test culture and process.
> We need to turn this corner as a group, we need to come together on some
> new behaviors, we need to generate large enough buy in to do something
> lasting.
>
> You all need to learn 'ant beast' (I'm fixing it to work *much* better)
> and my beasting test script gist (each has there place). I'm going to force
> it on you even if you don't ;)
>
>
> https://gist.githubusercontent.com/markrmiller/dbdb792216dc98b018ad/raw/cd084d01f405fa271af4b72a2848dc87374a1f71/gistfile1.sh
>
> For a long time I put most of my attention towards SolrCloud stability and
> scale. From 2012-2015 or so, that felt like the biggest threat to the
> project while also getting relatively little attention. I felt like it was
> 50/50 in 2014 that I'd even be talking to anyone about SolrCloud today.
> That hurdle is done and over with. The biggest threat to Solr and SolrCloud
> now is our tests.
>
> I'll be working from this umbrella issue:
>
> https://issues.apache.org/jira/browse/SOLR-12801 : Fix the tests, remove
> BadApples and AwaitsFix annotations, improve env for test development.
>
> Join the discussion.
>
> - Mark
> --
> - Mark
> about.me/markrmiller
>
>
>
Re: Solr Tests
Posted by Anshum Gupta <an...@apple.com>.
+1 Mark! I’m in.
Anshum
> On Oct 24, 2018, at 11:05 PM, Mark Miller <ma...@gmail.com> wrote:
>
> My boss has indicated I'm going to get a little time to spend on addressing our test situation. Turns out the current downward trend situation is a little worrying to some that count on the project down the road ;)
>
> We can finally dig out though, I promise.
>
> It's going to be a little bit before I'm clear and gathering steam, but I've got this figured out. I'm finally playing checkers instead of rock / paper / scissors.
>
> I'm going to eat a lot of work to get there and I promise I can get there, but I'm going to need some help.
>
> If you have an interest in changing our test situation, and I know some of you do, please join me as I invite others to help out more in a short time.
>
> Beyond some help with code effort though, please get involved in discussions and JIRA issues around changing our test culture and process. We need to turn this corner as a group, we need to come together on some new behaviors, we need to generate large enough buy in to do something lasting.
>
> You all need to learn 'ant beast' (I'm fixing it to work *much* better) and my beasting test script gist (each has there place). I'm going to force it on you even if you don't ;)
>
> https://gist.githubusercontent.com/markrmiller/dbdb792216dc98b018ad/raw/cd084d01f405fa271af4b72a2848dc87374a1f71/gistfile1.sh <https://gist.githubusercontent.com/markrmiller/dbdb792216dc98b018ad/raw/cd084d01f405fa271af4b72a2848dc87374a1f71/gistfile1.sh>
>
> For a long time I put most of my attention towards SolrCloud stability and scale. From 2012-2015 or so, that felt like the biggest threat to the project while also getting relatively little attention. I felt like it was 50/50 in 2014 that I'd even be talking to anyone about SolrCloud today. That hurdle is done and over with. The biggest threat to Solr and SolrCloud now is our tests.
>
> I'll be working from this umbrella issue:
>
> https://issues.apache.org/jira/browse/SOLR-12801 <https://issues.apache.org/jira/browse/SOLR-12801> : Fix the tests, remove BadApples and AwaitsFix annotations, improve env for test development.
>
> Join the discussion.
>
> - Mark
> --
> - Mark
> about.me/markrmiller <http://about.me/markrmiller>
Re: Solr Tests
Posted by Shawn Heisey <ap...@elyograg.org>.
On 10/25/2018 12:05 AM, Mark Miller wrote:
> If you have an interest in changing our test situation, and I know
> some of you do, please join me as I invite others to help out more in
> a short time.
I'm interested in helping.
There's a lot of code that I'm not at all familiar with, and I've found
that I get *very* lost when I try exploring the rabbit hole to try and
*become* familiar with the code. I learn a little more every day, which
mostly serves to inform me just how much I *don't* know.
If I see something I think I can do, I will work on it. If anybody sees
a suitable candidate that I miss, feel free to prod me towards it.
> Beyond some help with code effort though, please get involved in
> discussions and JIRA issues around changing our test culture and
> process. We need to turn this corner as a group, we need to come
> together on some new behaviors, we need to generate large enough buy
> in to do something lasting.
I will add my two cents, and hopefully I won't aggravate everyone TOO
much. :)
Thanks,
Shawn
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org