-Dtests.iters not working?

[Shai Erera <se...@gmail.com>]

Hi

I tried to run a test with -Dtests.iters=100 and it doesn't work:

-- from eclipse, nothing ran, not even once
-- from cmd-line, the test was run once

I did "svn up", then "ant clean clean-jars resolve eclipse" and tried
again, same results

Does it work for anyone?

Shai

Re: -Dtests.iters not working?

[Dawid Weiss <da...@cs.put.poznan.pl>]

> eclipse and it worked with -Dtests.iters. I wonder why it didn't work from
> cmd-line though, as testmethod is an alias to tests.method.

It should work with both.

> Isn't there a way to make running a single test-method still runnable with
> -Dtests.iters? This is something that used to work, how come that we lost

It did not work properly. It was a loop that rerun the test method but
didn't really play nice with rules, hooks etc. It's virtually as if
you put:

for (int i = 0; i < n; i++) {
 invokeTestMethod();
}

into another method and ran it as a test. The new option (that's part
of the reason it was named slightly differently) is a real JUnit
compatible test reiteration -- it picks the same or different seeds,
is repeatable, invokes hooks and rules, puts every reiteration into a
separate entry into reports, etc.

I consider this a step forward rather than backward but of course you
can argue otherwise.

> ensuring that at least what used to work doesn't stop working. And also try
> to avoid unnecessary changes like the tests.iter rename - it's something
> that was working and we got used to it, why change?

Explained above. It's not the same thing.

> Now I need to adapt the way I run tests in eclipse, by never running a
> single test-method, but always running classes with -Dtests.method. If this

You can run a single test method and it will work identically as
before as long as you don't run with multiple iterations. Multiple
iterations (and parameters, and different @Seed annotations) will
expand into different test names and Eclipse filters them out seeking
for an exact method name match.

Unfortunately this cannot be fixed or worked around, this is how JUnit
is internally designed. I consider this a flaw but pushing JUnit
changes is quite hard because of backcompat concerns.

> is something that you can fix, I'd appreciate if you can try.

No. This isn't something that I came up with overnight -- it's a
tedious trial and error of what work and what doesn't. This is the
best compromise I could find.

> Is tests.iter(s).min back working, or is it still disabled? This was a very
> useful feature too.

It doesn't exist anymore. Again -- because of how JUnit is designed
(tests are collected by the core before anything is run, then there is
no way to interrupt execution). I could probably provide a workaround
in which tests would be ignored after the first detected failure if
more people miss that feature (nobody was complaining so far).

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: -Dtests.iters not working?

[Shai Erera <se...@gmail.com>]

Thanks for the explanation.

The cmd-line I used is:

ant test -Dtestcase=TestDirectoryTaxonomyWriter
-Dtestmethod=testConcurrency -Dtests.iters=100

I see in your example that you use -Dtests.method, and I tried that in
eclipse and it worked with -Dtests.iters. I wonder why it didn't work from
cmd-line though, as testmethod is an alias to tests.method.

Isn't there a way to make running a single test-method still runnable with
-Dtests.iters? This is something that used to work, how come that we lost
it?

I think we need to be more careful in all the test-framework changes, by
ensuring that at least what used to work doesn't stop working. And also try
to avoid unnecessary changes like the tests.iter rename - it's something
that was working and we got used to it, why change?

Now I need to adapt the way I run tests in eclipse, by never running a
single test-method, but always running classes with -Dtests.method. If this
is something that you can fix, I'd appreciate if you can try. We don't need
to adopt every bit of JUnit, only the one that makes our life easier.

Is tests.iter(s).min back working, or is it still disabled? This was a very
useful feature too.

Shai

On Wed, May 16, 2012 at 4:31 PM, Dawid Weiss
<da...@cs.put.poznan.pl>wrote:

> > I tried to run a test with -Dtests.iters=100 and it doesn't work:
> >
> > -- from eclipse, nothing ran, not even once
>
> What was your test definition (what did you try to run)? If it's a
> single method then it's very likely that Eclipse filtered out all
> repetitions -- JUnit uses "test description filtering" that doesn't
> really comply with the notion of multiple executions of the same test
> method (read: every method must have a unique description, otherwise
> tools go crazy). RandomizedRunner expands reiterations of a given
> method into multiple descriptions, each with a unique name. If you're
> using a constant -Dtests.seed these may be filterable from the GUI but
> otherwise they'll be different tests each time and so the filter will
> remove all current method executions, leaving none.
>
> In short: run a class (suite), not a specific method. If you want to
> narrow down to a specific method, use filtering at the runner level,
> not Eclipse's. Attached is a screenshot of what it looks like
> (example).
>
> > -- from cmd-line, the test was run once
>
> What was your command line (exactly)?
>
> Dawid
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>

Re: -Dtests.iters not working?

[Dawid Weiss <da...@cs.put.poznan.pl>]

> I tried to run a test with -Dtests.iters=100 and it doesn't work:
>
> -- from eclipse, nothing ran, not even once

What was your test definition (what did you try to run)? If it's a
single method then it's very likely that Eclipse filtered out all
repetitions -- JUnit uses "test description filtering" that doesn't
really comply with the notion of multiple executions of the same test
method (read: every method must have a unique description, otherwise
tools go crazy). RandomizedRunner expands reiterations of a given
method into multiple descriptions, each with a unique name. If you're
using a constant -Dtests.seed these may be filterable from the GUI but
otherwise they'll be different tests each time and so the filter will
remove all current method executions, leaving none.

In short: run a class (suite), not a specific method. If you want to
narrow down to a specific method, use filtering at the runner level,
not Eclipse's. Attached is a screenshot of what it looks like
(example).

> -- from cmd-line, the test was run once

What was your command line (exactly)?

Dawid