You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Grant Ingersoll <gs...@apache.org> on 2011/12/08 14:36:41 UTC
Tests running time
MAHOUT-916 and 917 are attempts to address the running time of our tests. As Sean rightfully pointed out, there are probably opportunities to simply cut down the sizes of some of these tests w/o effecting there correctness. To that end, if people can take a look at:
https://builds.apache.org/job/Mahout-Quality/1237/testReport/junit/
You can get a sense as to which tests are taking a long time. The main culprits are:
1. Vectorizer
2. SSVD
3. K-Means
4. taste.hadoop.item
5. taste.hadoop.als
6. PFPGrowth
-Grant
--------------------------------------------
Grant Ingersoll
http://www.lucidimagination.com
Re: Tests running time
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
which brings me to the thought, we could probably employ some
heuristics to deduce that parameter by the time ABt job runs. or just
set it sufficiently high there just for the case of ABt. Some danger
is if we set it significantly >> s, then we may overallocate some
memory we will never use. but it probably will not be very
significant. I can probably take a look.
On Fri, Dec 9, 2011 at 11:30 AM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> I don't think it would make sense for SSVD to remove MR. I mean, sure,
> we can test something like Givens solver independently, but it would
> not be testing much really.
>
> I will reduce the dimensionality there.
>
> Also, there are a lot of tests ( sparse/dense, sparse with power
> iteration and without , dense with power iteration and without... ) so
> we probably could just nix some of those and assume they "work" by
> manual enabling and verification of a committer.
>
>
>
> On Fri, Dec 9, 2011 at 9:00 AM, Grant Ingersoll <gs...@apache.org> wrote:
>>
>> On Dec 8, 2011, at 12:55 PM, Sean Owen wrote:
>>
>>> This could well be it. While every Random everywhere gets initialized to a
>>> known initial state, at the start of every @Test method, you could get
>>> different sequences if other tests are in progress in parallel in the same
>>> JVM.
>>>
>>> Ideally tests aren't that sensitive to the sequence of random numbers -- if
>>> that's the case. And here it may well be the case.
>>>
>>> Can this be set to fork a JVM per test class? that would probably work.
>>
>> I'm no maven expert, but based on my reading of the docs and the things I've tried, it seems like "always" forking isn't going to get the parallelism we want. On the other hand, we can't seem to run in parallel w/ fork once due to some threading issues. What do others think?
>>
>> At the end of the day, I believe most of our performance issues are due to running full M/R jobs. So, we either rework them to just test mappers and reducers independently and move the long running full tests to nightly/weekly tests or we go off an improve local mode in Hadoop to give better performance.
>>
>> I'd vote for the former since it is the only one we are likely to get done reasonably soon.
>>
>>>
>>> On Thu, Dec 8, 2011 at 7:43 PM, Grant Ingersoll <gs...@apache.org> wrote:
>>>
>>>>
>>>> On Dec 8, 2011, at 2:39 PM, Grant Ingersoll wrote:
>>>>
>>>>>
>>>>> On Dec 8, 2011, at 2:23 PM, Grant Ingersoll wrote:
>>>>>
>>>>>> If I add parallel, fork always to the main surefire config, I get
>>>> failures all over the place for things like:
>>>>>> Failed tests:
>>>> testHebbianSolver(org.apache.mahout.math.decomposer.hebbian.TestHebbianSolver):
>>>> Error: {0.06146049974880152 too high! (for eigen 3)
>>>>>> consistency(org.apache.mahout.math.jet.random.NormalTest):
>>>> offset=0.000 scale=1.000 Z = 8.2
>>>>>> consistency(org.apache.mahout.math.jet.random.ExponentialTest):
>>>> offset=0.000 scale=100.000 Z = 8.7
>>>>>>
>>>>>
>>>>> Check that, it seems each run can produce different failures, which
>>>> leads me to believe we have some shared values in our tests
>>>>
>>>> Random.getRandom() the culprit, perhaps?
>>>>
>>>>>
>>>>>
>>>>>> All of these pass individually and when not in parallel for me.
>>>>>>
>>>>>> Here's my config:
>>>>>> <plugin>
>>>>>> <groupId>org.apache.maven.plugins</groupId>
>>>>>> <artifactId>maven-surefire-plugin</artifactId>
>>>>>> <version>2.11</version>
>>>>>> <configuration>
>>>>>> <parallel>classes</parallel>
>>>>>> <forkMode>always</forkMode>
>>>>>> <perCoreThreadCount>true</perCoreThreadCount>
>>>>>> </configuration>
>>>>>> </plugin>
>>>>>>
>>>>>> Anyone else seeing that?
>>>>>>
>>>>>>
>>>>>> On Dec 8, 2011, at 1:53 PM, Dmitriy Lyubimov wrote:
>>>>>>
>>>>>>> SSVD actually runs a rather small test but it is a MR job in local
>>>>>>> mode, there's nothing to cut down there in terms of size (not much
>>>>>>> anyway). It's just what it takes to initialize and run all jobs (and
>>>>>>> since it is local, it is also single threaded, so it actually runs V
>>>>>>> and U jobs sequentially instead of parallel so it's even longer
>>>>>>> because of that (4 jobs stringed all in all).
>>>>>>>
>>>>>>> But i will take a look, although even if i reduce solution size, it
>>>>>>> will still likely not reduce running time by more than 20%.
>>>>>>>
>>>>>>> On Thu, Dec 8, 2011 at 5:42 AM, David Murgatroyd <dm...@gmail.com>
>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Dec 8, 2011, at 8:36 AM, Grant Ingersoll <gs...@apache.org>
>>>> wrote:
>>>>>>>>
>>>>>>>>> MAHOUT-916 and 917 are attempts to address the running time of our
>>>> tests. As Sean rightfully pointed out, there are probably opportunities to
>>>> simply cut down the sizes of some of these tests w/o effecting there
>>>> correctness. To that end, if people can take a look at:
>>>>>>>>> https://builds.apache.org/job/Mahout-Quality/1237/testReport/junit/
>>>>>>>>>
>>>>>>>>> You can get a sense as to which tests are taking a long time. The
>>>> main culprits are:
>>>>>>>>> 1. Vectorizer
>>>>>>>>> 2. SSVD
>>>>>>>>> 3. K-Means
>>>>>>>>> 4. taste.hadoop.item
>>>>>>>>> 5. taste.hadoop.als
>>>>>>>>> 6. PFPGrowth
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -Grant
>>>>>>>>>
>>>>>>>>> --------------------------------------------
>>>>>>>>> Grant Ingersoll
>>>>>>>>> http://www.lucidimagination.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>> --------------------------------------------
>>>>>> Grant Ingersoll
>>>>>> http://www.lucidimagination.com
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --------------------------------------------
>>>>> Grant Ingersoll
>>>>> http://www.lucidimagination.com
>>>>>
>>>>>
>>>>>
>>>>
>>>> --------------------------------------------
>>>> Grant Ingersoll
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>>
>>>>
>>
>> --------------------------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com
>>
>>
>>
Re: Tests running time
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
I don't think it would make sense for SSVD to remove MR. I mean, sure,
we can test something like Givens solver independently, but it would
not be testing much really.
I will reduce the dimensionality there.
Also, there are a lot of tests ( sparse/dense, sparse with power
iteration and without , dense with power iteration and without... ) so
we probably could just nix some of those and assume they "work" by
manual enabling and verification of a committer.
On Fri, Dec 9, 2011 at 9:00 AM, Grant Ingersoll <gs...@apache.org> wrote:
>
> On Dec 8, 2011, at 12:55 PM, Sean Owen wrote:
>
>> This could well be it. While every Random everywhere gets initialized to a
>> known initial state, at the start of every @Test method, you could get
>> different sequences if other tests are in progress in parallel in the same
>> JVM.
>>
>> Ideally tests aren't that sensitive to the sequence of random numbers -- if
>> that's the case. And here it may well be the case.
>>
>> Can this be set to fork a JVM per test class? that would probably work.
>
> I'm no maven expert, but based on my reading of the docs and the things I've tried, it seems like "always" forking isn't going to get the parallelism we want. On the other hand, we can't seem to run in parallel w/ fork once due to some threading issues. What do others think?
>
> At the end of the day, I believe most of our performance issues are due to running full M/R jobs. So, we either rework them to just test mappers and reducers independently and move the long running full tests to nightly/weekly tests or we go off an improve local mode in Hadoop to give better performance.
>
> I'd vote for the former since it is the only one we are likely to get done reasonably soon.
>
>>
>> On Thu, Dec 8, 2011 at 7:43 PM, Grant Ingersoll <gs...@apache.org> wrote:
>>
>>>
>>> On Dec 8, 2011, at 2:39 PM, Grant Ingersoll wrote:
>>>
>>>>
>>>> On Dec 8, 2011, at 2:23 PM, Grant Ingersoll wrote:
>>>>
>>>>> If I add parallel, fork always to the main surefire config, I get
>>> failures all over the place for things like:
>>>>> Failed tests:
>>> testHebbianSolver(org.apache.mahout.math.decomposer.hebbian.TestHebbianSolver):
>>> Error: {0.06146049974880152 too high! (for eigen 3)
>>>>> consistency(org.apache.mahout.math.jet.random.NormalTest):
>>> offset=0.000 scale=1.000 Z = 8.2
>>>>> consistency(org.apache.mahout.math.jet.random.ExponentialTest):
>>> offset=0.000 scale=100.000 Z = 8.7
>>>>>
>>>>
>>>> Check that, it seems each run can produce different failures, which
>>> leads me to believe we have some shared values in our tests
>>>
>>> Random.getRandom() the culprit, perhaps?
>>>
>>>>
>>>>
>>>>> All of these pass individually and when not in parallel for me.
>>>>>
>>>>> Here's my config:
>>>>> <plugin>
>>>>> <groupId>org.apache.maven.plugins</groupId>
>>>>> <artifactId>maven-surefire-plugin</artifactId>
>>>>> <version>2.11</version>
>>>>> <configuration>
>>>>> <parallel>classes</parallel>
>>>>> <forkMode>always</forkMode>
>>>>> <perCoreThreadCount>true</perCoreThreadCount>
>>>>> </configuration>
>>>>> </plugin>
>>>>>
>>>>> Anyone else seeing that?
>>>>>
>>>>>
>>>>> On Dec 8, 2011, at 1:53 PM, Dmitriy Lyubimov wrote:
>>>>>
>>>>>> SSVD actually runs a rather small test but it is a MR job in local
>>>>>> mode, there's nothing to cut down there in terms of size (not much
>>>>>> anyway). It's just what it takes to initialize and run all jobs (and
>>>>>> since it is local, it is also single threaded, so it actually runs V
>>>>>> and U jobs sequentially instead of parallel so it's even longer
>>>>>> because of that (4 jobs stringed all in all).
>>>>>>
>>>>>> But i will take a look, although even if i reduce solution size, it
>>>>>> will still likely not reduce running time by more than 20%.
>>>>>>
>>>>>> On Thu, Dec 8, 2011 at 5:42 AM, David Murgatroyd <dm...@gmail.com>
>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Dec 8, 2011, at 8:36 AM, Grant Ingersoll <gs...@apache.org>
>>> wrote:
>>>>>>>
>>>>>>>> MAHOUT-916 and 917 are attempts to address the running time of our
>>> tests. As Sean rightfully pointed out, there are probably opportunities to
>>> simply cut down the sizes of some of these tests w/o effecting there
>>> correctness. To that end, if people can take a look at:
>>>>>>>> https://builds.apache.org/job/Mahout-Quality/1237/testReport/junit/
>>>>>>>>
>>>>>>>> You can get a sense as to which tests are taking a long time. The
>>> main culprits are:
>>>>>>>> 1. Vectorizer
>>>>>>>> 2. SSVD
>>>>>>>> 3. K-Means
>>>>>>>> 4. taste.hadoop.item
>>>>>>>> 5. taste.hadoop.als
>>>>>>>> 6. PFPGrowth
>>>>>>>>
>>>>>>>>
>>>>>>>> -Grant
>>>>>>>>
>>>>>>>> --------------------------------------------
>>>>>>>> Grant Ingersoll
>>>>>>>> http://www.lucidimagination.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>
>>>>> --------------------------------------------
>>>>> Grant Ingersoll
>>>>> http://www.lucidimagination.com
>>>>>
>>>>>
>>>>>
>>>>
>>>> --------------------------------------------
>>>> Grant Ingersoll
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>>
>>>
>>> --------------------------------------------
>>> Grant Ingersoll
>>> http://www.lucidimagination.com
>>>
>>>
>>>
>>>
>
> --------------------------------------------
> Grant Ingersoll
> http://www.lucidimagination.com
>
>
>
Re: Tests running time
Posted by Grant Ingersoll <gs...@apache.org>.
On Dec 8, 2011, at 12:55 PM, Sean Owen wrote:
> This could well be it. While every Random everywhere gets initialized to a
> known initial state, at the start of every @Test method, you could get
> different sequences if other tests are in progress in parallel in the same
> JVM.
>
> Ideally tests aren't that sensitive to the sequence of random numbers -- if
> that's the case. And here it may well be the case.
>
> Can this be set to fork a JVM per test class? that would probably work.
I'm no maven expert, but based on my reading of the docs and the things I've tried, it seems like "always" forking isn't going to get the parallelism we want. On the other hand, we can't seem to run in parallel w/ fork once due to some threading issues. What do others think?
At the end of the day, I believe most of our performance issues are due to running full M/R jobs. So, we either rework them to just test mappers and reducers independently and move the long running full tests to nightly/weekly tests or we go off an improve local mode in Hadoop to give better performance.
I'd vote for the former since it is the only one we are likely to get done reasonably soon.
>
> On Thu, Dec 8, 2011 at 7:43 PM, Grant Ingersoll <gs...@apache.org> wrote:
>
>>
>> On Dec 8, 2011, at 2:39 PM, Grant Ingersoll wrote:
>>
>>>
>>> On Dec 8, 2011, at 2:23 PM, Grant Ingersoll wrote:
>>>
>>>> If I add parallel, fork always to the main surefire config, I get
>> failures all over the place for things like:
>>>> Failed tests:
>> testHebbianSolver(org.apache.mahout.math.decomposer.hebbian.TestHebbianSolver):
>> Error: {0.06146049974880152 too high! (for eigen 3)
>>>> consistency(org.apache.mahout.math.jet.random.NormalTest):
>> offset=0.000 scale=1.000 Z = 8.2
>>>> consistency(org.apache.mahout.math.jet.random.ExponentialTest):
>> offset=0.000 scale=100.000 Z = 8.7
>>>>
>>>
>>> Check that, it seems each run can produce different failures, which
>> leads me to believe we have some shared values in our tests
>>
>> Random.getRandom() the culprit, perhaps?
>>
>>>
>>>
>>>> All of these pass individually and when not in parallel for me.
>>>>
>>>> Here's my config:
>>>> <plugin>
>>>> <groupId>org.apache.maven.plugins</groupId>
>>>> <artifactId>maven-surefire-plugin</artifactId>
>>>> <version>2.11</version>
>>>> <configuration>
>>>> <parallel>classes</parallel>
>>>> <forkMode>always</forkMode>
>>>> <perCoreThreadCount>true</perCoreThreadCount>
>>>> </configuration>
>>>> </plugin>
>>>>
>>>> Anyone else seeing that?
>>>>
>>>>
>>>> On Dec 8, 2011, at 1:53 PM, Dmitriy Lyubimov wrote:
>>>>
>>>>> SSVD actually runs a rather small test but it is a MR job in local
>>>>> mode, there's nothing to cut down there in terms of size (not much
>>>>> anyway). It's just what it takes to initialize and run all jobs (and
>>>>> since it is local, it is also single threaded, so it actually runs V
>>>>> and U jobs sequentially instead of parallel so it's even longer
>>>>> because of that (4 jobs stringed all in all).
>>>>>
>>>>> But i will take a look, although even if i reduce solution size, it
>>>>> will still likely not reduce running time by more than 20%.
>>>>>
>>>>> On Thu, Dec 8, 2011 at 5:42 AM, David Murgatroyd <dm...@gmail.com>
>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Dec 8, 2011, at 8:36 AM, Grant Ingersoll <gs...@apache.org>
>> wrote:
>>>>>>
>>>>>>> MAHOUT-916 and 917 are attempts to address the running time of our
>> tests. As Sean rightfully pointed out, there are probably opportunities to
>> simply cut down the sizes of some of these tests w/o effecting there
>> correctness. To that end, if people can take a look at:
>>>>>>> https://builds.apache.org/job/Mahout-Quality/1237/testReport/junit/
>>>>>>>
>>>>>>> You can get a sense as to which tests are taking a long time. The
>> main culprits are:
>>>>>>> 1. Vectorizer
>>>>>>> 2. SSVD
>>>>>>> 3. K-Means
>>>>>>> 4. taste.hadoop.item
>>>>>>> 5. taste.hadoop.als
>>>>>>> 6. PFPGrowth
>>>>>>>
>>>>>>>
>>>>>>> -Grant
>>>>>>>
>>>>>>> --------------------------------------------
>>>>>>> Grant Ingersoll
>>>>>>> http://www.lucidimagination.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>
>>>> --------------------------------------------
>>>> Grant Ingersoll
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>>
>>>
>>> --------------------------------------------
>>> Grant Ingersoll
>>> http://www.lucidimagination.com
>>>
>>>
>>>
>>
>> --------------------------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com
>>
>>
>>
>>
--------------------------------------------
Grant Ingersoll
http://www.lucidimagination.com
Re: Tests running time
Posted by Lance Norskog <go...@gmail.com>.
On Thu, Dec 8, 2011 at 3:52 PM, Sean Owen <sr...@gmail.com> wrote:
>
> On Thu, Dec 8, 2011 at 11:34 PM, Lance Norskog <go...@gmail.com> wrote:
>
> > A lot of tests are dependent on fixed random numbers.
> >
>
> (Which is not on purpose, I assume.)
>
They are coded that way- a great many unit tests of stochastic things do this:
assertEquals(0.2933927293472, object.call(), EPSILON);
The user ran the test by hand and hard-coded the test to match that value.
> > Starting a new JVM per test is a non-starter on smaller machines. Can
> > single-process mode be a default with a parameter for running parallel?
> >
>
> Why? this runs fine on my laptop.
>
I'm very happy for you. (A friend's wonderful catch-all rejoinder.)
--
Lance Norskog
goksron@gmail.com
Re: Tests running time
Posted by Sean Owen <sr...@gmail.com>.
On Thu, Dec 8, 2011 at 11:34 PM, Lance Norskog <go...@gmail.com> wrote:
> A lot of tests are dependent on fixed random numbers.
>
(Which is not on purpose, I assume.)
>
> Starting a new JVM per test is a non-starter on smaller machines. Can
> single-process mode be a default with a parameter for running parallel?
>
Why? this runs fine on my laptop.
>
> Or could the unit test framework have a Random factory object for each
> test?
>
It already does!
Re: Tests running time
Posted by Lance Norskog <go...@gmail.com>.
A lot of tests are dependent on fixed random numbers.
Starting a new JVM per test is a non-starter on smaller machines. Can
single-process mode be a default with a parameter for running parallel?
Or could the unit test framework have a Random factory object for each test?
On Thu, Dec 8, 2011 at 12:05 PM, Grant Ingersoll <gs...@apache.org>wrote:
> Progress! I had configured the surefire plugin in the wrong place
>
>
> On Dec 8, 2011, at 2:55 PM, Sean Owen wrote:
>
> > This could well be it. While every Random everywhere gets initialized to
> a
> > known initial state, at the start of every @Test method, you could get
> > different sequences if other tests are in progress in parallel in the
> same
> > JVM.
> >
> > Ideally tests aren't that sensitive to the sequence of random numbers --
> if
> > that's the case. And here it may well be the case.
> >
> > Can this be set to fork a JVM per test class? that would probably work.
> >
> > On Thu, Dec 8, 2011 at 7:43 PM, Grant Ingersoll <gs...@apache.org>
> wrote:
> >
> >>
> >> On Dec 8, 2011, at 2:39 PM, Grant Ingersoll wrote:
> >>
> >>>
> >>> On Dec 8, 2011, at 2:23 PM, Grant Ingersoll wrote:
> >>>
> >>>> If I add parallel, fork always to the main surefire config, I get
> >> failures all over the place for things like:
> >>>> Failed tests:
> >>
> testHebbianSolver(org.apache.mahout.math.decomposer.hebbian.TestHebbianSolver):
> >> Error: {0.06146049974880152 too high! (for eigen 3)
> >>>> consistency(org.apache.mahout.math.jet.random.NormalTest):
> >> offset=0.000 scale=1.000 Z = 8.2
> >>>> consistency(org.apache.mahout.math.jet.random.ExponentialTest):
> >> offset=0.000 scale=100.000 Z = 8.7
> >>>>
> >>>
> >>> Check that, it seems each run can produce different failures, which
> >> leads me to believe we have some shared values in our tests
> >>
> >> Random.getRandom() the culprit, perhaps?
> >>
> >>>
> >>>
> >>>> All of these pass individually and when not in parallel for me.
> >>>>
> >>>> Here's my config:
> >>>> <plugin>
> >>>> <groupId>org.apache.maven.plugins</groupId>
> >>>> <artifactId>maven-surefire-plugin</artifactId>
> >>>> <version>2.11</version>
> >>>> <configuration>
> >>>> <parallel>classes</parallel>
> >>>> <forkMode>always</forkMode>
> >>>> <perCoreThreadCount>true</perCoreThreadCount>
> >>>> </configuration>
> >>>> </plugin>
> >>>>
> >>>> Anyone else seeing that?
> >>>>
> >>>>
> >>>> On Dec 8, 2011, at 1:53 PM, Dmitriy Lyubimov wrote:
> >>>>
> >>>>> SSVD actually runs a rather small test but it is a MR job in local
> >>>>> mode, there's nothing to cut down there in terms of size (not much
> >>>>> anyway). It's just what it takes to initialize and run all jobs (and
> >>>>> since it is local, it is also single threaded, so it actually runs V
> >>>>> and U jobs sequentially instead of parallel so it's even longer
> >>>>> because of that (4 jobs stringed all in all).
> >>>>>
> >>>>> But i will take a look, although even if i reduce solution size, it
> >>>>> will still likely not reduce running time by more than 20%.
> >>>>>
> >>>>> On Thu, Dec 8, 2011 at 5:42 AM, David Murgatroyd <dm...@gmail.com>
> >> wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Dec 8, 2011, at 8:36 AM, Grant Ingersoll <gs...@apache.org>
> >> wrote:
> >>>>>>
> >>>>>>> MAHOUT-916 and 917 are attempts to address the running time of our
> >> tests. As Sean rightfully pointed out, there are probably
> opportunities to
> >> simply cut down the sizes of some of these tests w/o effecting there
> >> correctness. To that end, if people can take a look at:
> >>>>>>>
> https://builds.apache.org/job/Mahout-Quality/1237/testReport/junit/
> >>>>>>>
> >>>>>>> You can get a sense as to which tests are taking a long time. The
> >> main culprits are:
> >>>>>>> 1. Vectorizer
> >>>>>>> 2. SSVD
> >>>>>>> 3. K-Means
> >>>>>>> 4. taste.hadoop.item
> >>>>>>> 5. taste.hadoop.als
> >>>>>>> 6. PFPGrowth
> >>>>>>>
> >>>>>>>
> >>>>>>> -Grant
> >>>>>>>
> >>>>>>> --------------------------------------------
> >>>>>>> Grant Ingersoll
> >>>>>>> http://www.lucidimagination.com
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>
> >>>> --------------------------------------------
> >>>> Grant Ingersoll
> >>>> http://www.lucidimagination.com
> >>>>
> >>>>
> >>>>
> >>>
> >>> --------------------------------------------
> >>> Grant Ingersoll
> >>> http://www.lucidimagination.com
> >>>
> >>>
> >>>
> >>
> >> --------------------------------------------
> >> Grant Ingersoll
> >> http://www.lucidimagination.com
> >>
> >>
> >>
> >>
>
> --------------------------------------------
> Grant Ingersoll
> http://www.lucidimagination.com
>
>
>
>
--
Lance Norskog
goksron@gmail.com
Re: Tests running time
Posted by Sean Owen <sr...@gmail.com>.
On Mon, Dec 12, 2011 at 1:15 PM, Grant Ingersoll <gs...@apache.org>wrote:
> I'm not sure if it is completely valid, but it seems to me that if our
> tests can't run concurrently, it also raises a doubt as to whether some of
> our classes can be run concurrently.
>
I don't think there's a cause for concern; it's really an issue with how
the tests proceed, not the code they test. That is, we don't have a problem
with stuff returning a wrong answer or failing when tested concurrently,
it's that the tests expect something that they probably shouldn't, and that
changes with multiple threads.
>
> I think if we adopt the Lucene test framework (it's a separate JAR), we
> can get both the -Dtestseed thing and annotations such as @slow, @nightly,
> @weekly, etc. There are probably other useful nuggets in there too. It
> should work with Maven.
>
>
All that's needed is a way to (1) run only tests with a certain tag, and
then (2) in parallel run in separate JVMs the tests for each tag.
If you can figure out (2), you've already got a solution, I imagine: use
this to run core/examples/math/integration separately. The slightly nicer
thing is that the total runtime of each is probably more balanced than just
the slows, just the nightly, etc. No tagging needed.
Re: Tests running time
Posted by Grant Ingersoll <gs...@apache.org>.
On Dec 12, 2011, at 7:11 AM, Sean Owen wrote:
> On Mon, Dec 12, 2011 at 11:59 AM, Grant Ingersoll <gs...@apache.org>wrote:
>
>> In Lucene, we simply print out what the seed is if the tests fail and then
>> you can rerun that test by saying ant -Dtestseed=XXXX test
>>
>
> I like that -- it's a separate thing but it's a fine idea too. It lets you
> at least try different seeds while not sacrificing repeatability.
>
>
>
>> AFAICT, the issue with some of these tests, LogLikelihoodTest (the
>> frequency comparison test fails when you change the seed) is that they are
>> testing specific values as outcomes based on a specific test seed. That's
>> OK, but it doesn't lend itself to the reset problem and it makes things
>> problematic to debug should we ever change the seed, etc.
>>
>
> Yes that's right, and so it only works if it sees the same random number
> sequence. Even if it didn't we'd still want repeatability, though ideally
> it would work with all random seeds so that we can use different seeds.
>
> Assertions like this are good for detecting changing behavior, but yes not
> strictly asserting something that must not be false.
>
> The easiest solution is to just remove assertions that seem to be testing
> the random number sequence; a slightly better idea would be to relax them
> to assert things that must be true (e.g. count must be positive and less
> than the number of items or something).
>
+1. We do seem to be testing the RNG in some of these tests a bit more than we are testing the actual thing.
> Better still would be to come up with assertions that ought to be true with
> very high probability, such that a failure is almost certainly a symptom of
> a bug. It's hard to write those bounds; you could guess conservatively and
> then relax as you observe failures that don't appear to be a bug. This is a
> fair bit of work though!
>
I'm not sure if it is completely valid, but it seems to me that if our tests can't run concurrently, it also raises a doubt as to whether some of our classes can be run concurrently.
>>
>> This is where Ant seems to be a lot better.
>
>
> Right, it would be fairly easy to fork JVMs in Ant. I imagine there's some
> way to do it Maven, but how I don't know.
>
> Hmm, is there any easy way to have it run tests in all of the modules,
> independently and in parallel, in n JVMs? That would do it too.
> Or, say we had a separate test src dir for "big" tests -- can that
> secondary test target be told to run in parallel?
> I suppose you could write an Ant target to do the work and then invoke it
> in Maven.
I think if we adopt the Lucene test framework (it's a separate JAR), we can get both the -Dtestseed thing and annotations such as @slow, @nightly, @weekly, etc. There are probably other useful nuggets in there too. It should work with Maven.
Re: Tests running time
Posted by Sean Owen <sr...@gmail.com>.
On Mon, Dec 12, 2011 at 11:59 AM, Grant Ingersoll <gs...@apache.org>wrote:
> In Lucene, we simply print out what the seed is if the tests fail and then
> you can rerun that test by saying ant -Dtestseed=XXXX test
>
I like that -- it's a separate thing but it's a fine idea too. It lets you
at least try different seeds while not sacrificing repeatability.
> AFAICT, the issue with some of these tests, LogLikelihoodTest (the
> frequency comparison test fails when you change the seed) is that they are
> testing specific values as outcomes based on a specific test seed. That's
> OK, but it doesn't lend itself to the reset problem and it makes things
> problematic to debug should we ever change the seed, etc.
>
Yes that's right, and so it only works if it sees the same random number
sequence. Even if it didn't we'd still want repeatability, though ideally
it would work with all random seeds so that we can use different seeds.
Assertions like this are good for detecting changing behavior, but yes not
strictly asserting something that must not be false.
The easiest solution is to just remove assertions that seem to be testing
the random number sequence; a slightly better idea would be to relax them
to assert things that must be true (e.g. count must be positive and less
than the number of items or something).
Better still would be to come up with assertions that ought to be true with
very high probability, such that a failure is almost certainly a symptom of
a bug. It's hard to write those bounds; you could guess conservatively and
then relax as you observe failures that don't appear to be a bug. This is a
fair bit of work though!
>
> >
> > To be clear: the injection is not at all the issue here. I'm happy to
> write
> > an essay on this, but, the punchline is: there is virtually no way to
> have
> > deterministic tests in one JVM while running all the tests in parallel.
> We
> > were pursuing means to execute subsets of the tests in parallel in
> > different JVMs, which can work and does work. We haven't yet found a way
> to
> > do it in Maven.
>
> This is where Ant seems to be a lot better.
Right, it would be fairly easy to fork JVMs in Ant. I imagine there's some
way to do it Maven, but how I don't know.
Hmm, is there any easy way to have it run tests in all of the modules,
independently and in parallel, in n JVMs? That would do it too.
Or, say we had a separate test src dir for "big" tests -- can that
secondary test target be told to run in parallel?
I suppose you could write an Ant target to do the work and then invoke it
in Maven.
Re: Tests running time
Posted by Grant Ingersoll <gs...@apache.org>.
On Dec 11, 2011, at 2:42 PM, Sean Owen wrote:
> On Sun, Dec 11, 2011 at 7:35 PM, Ted Dunning <te...@gmail.com> wrote:
>
>> The right way to handle this is to have instances get a random number
>> generator that works like it should. Magic resets in the middle of
>> operation are not a good idea.
>>
>
> Why would the caller care? It's all random numbers, whether "reset" or not.
>
>
>>
>> I think we need a better way to inject generators that doesn't involve
>> statics.
>>
>>
> There is no way to control and reset Randoms other than what's done in
> RandomWrapper (with statics). If you think deterministic tests are a good
> goal, there's no argument with this, as it's the only way to do it. (And if
> you don't agree that's another conversation -- but we do need this.)
In Lucene, we simply print out what the seed is if the tests fail and then you can rerun that test by saying ant -Dtestseed=XXXX test
AFAICT, the issue with some of these tests, LogLikelihoodTest (the frequency comparison test fails when you change the seed) is that they are testing specific values as outcomes based on a specific test seed. That's OK, but it doesn't lend itself to the reset problem and it makes things problematic to debug should we ever change the seed, etc.
>
> To be clear: the injection is not at all the issue here. I'm happy to write
> an essay on this, but, the punchline is: there is virtually no way to have
> deterministic tests in one JVM while running all the tests in parallel. We
> were pursuing means to execute subsets of the tests in parallel in
> different JVMs, which can work and does work. We haven't yet found a way to
> do it in Maven.
This is where Ant seems to be a lot better.
Re: Tests running time
Posted by Grant Ingersoll <gs...@apache.org>.
On Dec 12, 2011, at 2:05 AM, Sean Owen wrote:
>
> It *seems* so much more like a test issue to me, solvable in the test
> harness, and in a clear way: just split tests n ways across n JVMs instead
> of 1 JVM with n threads. No (further) reliance on code being exactly well
> behaved. It's just we don't have a one-liner to do that. Compared with
> seasoning the code with ThreadLocal, and its associated small overhead, and
> all the work to do that surgery, just feels very much like the hard way to
> solve this.
Perhaps this will work, but the hard part is Maven doesn't offer batch granularity. It's either once or always for forking. Using always + parallel seems to yield no benefit in terms of performance.
Re: Tests running time
Posted by Sean Owen <sr...@gmail.com>.
On Mon, Dec 12, 2011 at 2:49 AM, Ted Dunning <te...@gmail.com> wrote:
> > > Why would the caller care? It's all random numbers, whether "reset" or
> > not.
> >
>
> The care is about determinism.
>
Completely agree, it's the tests that care, not the caller itself.
> If the RandUtils notes that the current thread stated we are in test mode
> before asking for a random number generator then it can return a random
> generator with the test seed. Otherwise, it can return a more robustly
> seeded generator. The generators in separate test threads are independent
> objects with independent state.
>
>
Just noting the thread is in test mode doesn't do the trick -- the code
already does what you describe.
Are you talking about ThreadLocal? That is nearly the solution, though it
still involves static members. The performance overhead of using
ThreadLocal is probably quite small here, no problem. It does still need
code surgery, as any Random potentially needs to be ThreadLocal<Random>
unless you can know it is never shared. And any computation resulting from
these can not be shared, likewise, or else the result changes in
potentially nondeterministic ways in the presence of multiple threads. The
knock-on effect is, I would guess, significant.
In this regard I think it's the same challenge you get without ThreadLocal,
but at least it gives you a way to share a Random across objects without
having to share it across threads. I do agree this is all possible in
principle, just not in practice, here. My guess is that there will be some
shared state that just can't be un-shared without a lot of work, or so much
more change (many more ThreadLocals) that it is unpalatable. And of course,
it just takes one new bit of code not doing this all right to make tests
non-deterministic again -- you can police it but even "new Random()" is
being written in new code today
My guess is this is "virtually impossible" but that is just a guess --
brave souls welcome to try a patch after noting the above and see how it
flies.
It *seems* so much more like a test issue to me, solvable in the test
harness, and in a clear way: just split tests n ways across n JVMs instead
of 1 JVM with n threads. No (further) reliance on code being exactly well
behaved. It's just we don't have a one-liner to do that. Compared with
seasoning the code with ThreadLocal, and its associated small overhead, and
all the work to do that surgery, just feels very much like the hard way to
solve this.
Re: Tests running time
Posted by Ted Dunning <te...@gmail.com>.
On Sun, Dec 11, 2011 at 6:48 PM, Lance Norskog <go...@gmail.com> wrote:
> What about using ThreadLocal generators?
>
> On Sun, Dec 11, 2011 at 11:42 AM, Sean Owen <sr...@gmail.com> wrote:
> > On Sun, Dec 11, 2011 at 7:35 PM, Ted Dunning <te...@gmail.com>
> wrote:
> >
> >> The right way to handle this is to have instances get a random number
> >> generator that works like it should. Magic resets in the middle of
> >> operation are not a good idea.
> >>
> >
> > Why would the caller care? It's all random numbers, whether "reset" or
> not.
>
The care is about determinism.
Tests are easier if they produce the same result every time. Some tests
are probabilistic in nature and thus can't be guaranteed to pass every time
without making them so lax as to be not useful. By making them
deterministic, they become regression tests rather than tests in principle,
but also become more useful in general.
>> I think we need a better way to inject generators that doesn't involve
>> statics.
>>
>>
> There is no way to control and reset Randoms other than what's done in
> RandomWrapper (with statics). If you think deterministic tests are a good
> goal, there's no argument with this, as it's the only way to do it. (And
if
> you don't agree that's another conversation -- but we do need this.)
>
> To be clear: the injection is not at all the issue here. I'm happy to
write
> an essay on this, but, the punchline is: there is virtually no way to have
> deterministic tests in one JVM while running all the tests in parallel.
This is just simply not true.
You absolutely can have deterministic and independent generators.
If the RandUtils notes that the current thread stated we are in test mode
before asking for a random number generator then it can return a random
generator with the test seed. Otherwise, it can return a more robustly
seeded generator. The generators in separate test threads are independent
objects with independent state.
> We were pursuing means to execute subsets of the tests in parallel in
> different JVMs, which can work and does work. We haven't yet found a way
to
> do it in Maven.
But this isn't necessary. We can just fix RandUtils and be on our way.
Re: Tests running time
Posted by Lance Norskog <go...@gmail.com>.
What about using ThreadLocal generators?
On Sun, Dec 11, 2011 at 11:42 AM, Sean Owen <sr...@gmail.com> wrote:
> On Sun, Dec 11, 2011 at 7:35 PM, Ted Dunning <te...@gmail.com> wrote:
>
>> The right way to handle this is to have instances get a random number
>> generator that works like it should. Magic resets in the middle of
>> operation are not a good idea.
>>
>
> Why would the caller care? It's all random numbers, whether "reset" or not.
>
>
>>
>> I think we need a better way to inject generators that doesn't involve
>> statics.
>>
>>
> There is no way to control and reset Randoms other than what's done in
> RandomWrapper (with statics). If you think deterministic tests are a good
> goal, there's no argument with this, as it's the only way to do it. (And if
> you don't agree that's another conversation -- but we do need this.)
>
> To be clear: the injection is not at all the issue here. I'm happy to write
> an essay on this, but, the punchline is: there is virtually no way to have
> deterministic tests in one JVM while running all the tests in parallel. We
> were pursuing means to execute subsets of the tests in parallel in
> different JVMs, which can work and does work. We haven't yet found a way to
> do it in Maven.
--
Lance Norskog
goksron@gmail.com
Re: Tests running time
Posted by Sean Owen <sr...@gmail.com>.
On Sun, Dec 11, 2011 at 7:35 PM, Ted Dunning <te...@gmail.com> wrote:
> The right way to handle this is to have instances get a random number
> generator that works like it should. Magic resets in the middle of
> operation are not a good idea.
>
Why would the caller care? It's all random numbers, whether "reset" or not.
>
> I think we need a better way to inject generators that doesn't involve
> statics.
>
>
There is no way to control and reset Randoms other than what's done in
RandomWrapper (with statics). If you think deterministic tests are a good
goal, there's no argument with this, as it's the only way to do it. (And if
you don't agree that's another conversation -- but we do need this.)
To be clear: the injection is not at all the issue here. I'm happy to write
an essay on this, but, the punchline is: there is virtually no way to have
deterministic tests in one JVM while running all the tests in parallel. We
were pursuing means to execute subsets of the tests in parallel in
different JVMs, which can work and does work. We haven't yet found a way to
do it in Maven.
Re: Tests running time
Posted by Ted Dunning <te...@gmail.com>.
The right way to handle this is to have instances get a random number
generator that works like it should. Magic resets in the middle of
operation are not a good idea.
I think we need a better way to inject generators that doesn't involve
statics.
On Sun, Dec 11, 2011 at 6:24 AM, Sean Owen <sr...@gmail.com> wrote:
> Yes that's exactly what's happening -- not why the tests aren't running
> fast, but why running them in parallel in one JVM results in
> non-deterministic results.
>
> If by "not use statics" you mean hold a static reference to a Random in
> client code, yes, that could help, except that you'd also have to not share
> objects within a test. And of course it's possible a random value affects
> initialization of another, shared static data structure. And, making a
> Random non-static may be infeasible: imagine a class which is instantiated
> a billion times and shares one RNG; giving each instance its own RNG would
> dramatically increase memory use and init time. And you would never be able
> all non-determinism: consider third-party code we use. I don't know if it's
> feasible unfortunately.
>
> The point of resetting the RNGs is to (try to) make the tests repeatable.
> If a test fails 1 in 1000 times due to some sequence of random numbers and
> can't be reproduced we have a hard time tracking it down. You have to reset
> potentially every RNG to a known state, not just ones that are instantiated
> later, or else the purpose is defeated.
>
> I think it's coincidence that it passes with a different RNG set-up; the
> test expects the RNG to not be reset in the middle (as it works in serial
> execution only) and in parallel execution, when the reset is turned off, it
> doesn't. Of course it could fall the other way -- parallel execution causes
> some shared RNG to generate a different sequence, and that's what in theory
> we're trying to dodge, since it's not repeatable.
>
>
> On Sun, Dec 11, 2011 at 1:08 PM, Grant Ingersoll <gsingers@apache.org
> >wrote:
>
> > In working through what I _think_ will be the primary viable way to make
> > this stuff faster (parallel execution, fork once) it appears to me that
> the
> > primary concurrency issue is due to how we initialize the test seed and
> the
> > fact that we loop over all RandomWrapper objects and reset them. So,
> it's
> > likely the case that in mid stream of some of the tests, the RNG is
> getting
> > reset by other calls to the static useTestSeed() method.
> >
> > Of course, there might be other concurrency issues beyond that, but this
> > seems like the most likely one to start. Thus, the question is how to
> fix
> > it. The obvious one, I suppose, is to not use statics for this stuff.
> > Another is, to perhaps, use a system property (-DuseTestSeed=true and/or
> > -DuseSeed=<SEED>, the latter being useful for debugging other things)
> that
> > is set upon invocation in the test plugin, but has the downside that it
> > would also need to be set when running from an IDE.
> >
> > And, to Sean's point below, it seems that we may have some test
> > dependencies on the specific set of random numbers and the outcomes they
> > produce.
> >
> > Thoughts? Other ideas?
> >
> >
>
Re: Tests running time
Posted by Sean Owen <sr...@gmail.com>.
Yes that's exactly what's happening -- not why the tests aren't running
fast, but why running them in parallel in one JVM results in
non-deterministic results.
If by "not use statics" you mean hold a static reference to a Random in
client code, yes, that could help, except that you'd also have to not share
objects within a test. And of course it's possible a random value affects
initialization of another, shared static data structure. And, making a
Random non-static may be infeasible: imagine a class which is instantiated
a billion times and shares one RNG; giving each instance its own RNG would
dramatically increase memory use and init time. And you would never be able
all non-determinism: consider third-party code we use. I don't know if it's
feasible unfortunately.
The point of resetting the RNGs is to (try to) make the tests repeatable.
If a test fails 1 in 1000 times due to some sequence of random numbers and
can't be reproduced we have a hard time tracking it down. You have to reset
potentially every RNG to a known state, not just ones that are instantiated
later, or else the purpose is defeated.
I think it's coincidence that it passes with a different RNG set-up; the
test expects the RNG to not be reset in the middle (as it works in serial
execution only) and in parallel execution, when the reset is turned off, it
doesn't. Of course it could fall the other way -- parallel execution causes
some shared RNG to generate a different sequence, and that's what in theory
we're trying to dodge, since it's not repeatable.
On Sun, Dec 11, 2011 at 1:08 PM, Grant Ingersoll <gs...@apache.org>wrote:
> In working through what I _think_ will be the primary viable way to make
> this stuff faster (parallel execution, fork once) it appears to me that the
> primary concurrency issue is due to how we initialize the test seed and the
> fact that we loop over all RandomWrapper objects and reset them. So, it's
> likely the case that in mid stream of some of the tests, the RNG is getting
> reset by other calls to the static useTestSeed() method.
>
> Of course, there might be other concurrency issues beyond that, but this
> seems like the most likely one to start. Thus, the question is how to fix
> it. The obvious one, I suppose, is to not use statics for this stuff.
> Another is, to perhaps, use a system property (-DuseTestSeed=true and/or
> -DuseSeed=<SEED>, the latter being useful for debugging other things) that
> is set upon invocation in the test plugin, but has the downside that it
> would also need to be set when running from an IDE.
>
> And, to Sean's point below, it seems that we may have some test
> dependencies on the specific set of random numbers and the outcomes they
> produce.
>
> Thoughts? Other ideas?
>
>
Re: Tests running time
Posted by Grant Ingersoll <gs...@apache.org>.
As a point of reference, if I comment out the reset() code in useTestSeed for the math package, all tests pass w/ parallel execution and fork once. Of course, that's just one piece.
I guess I don't understand why we need to do all that reset stuff there anyway. If you are using the test seed, it gets marked in the @Before (and probably should be @BeforeClass) and then that test should have all of it's RandomWrapper's set to the test seed going forward. Why do we need to track all the other randoms that were ever created and reset them?
This seems to have been introduced on 2/19/10 by:
<snip>
Add several more unit tests for cf.taste. Make random numbers all but guaranteed to be deterministic during unit tests, to allow for repeatable tests of components with randomness.
git-svn-id: https://svn.apache.org/repos/asf/lucene/mahout/trunk@911810 13f79535-47bb-0310-9956-ffa450edef68
</snip>
-Grant
On Dec 11, 2011, at 6:08 AM, Grant Ingersoll wrote:
> In working through what I _think_ will be the primary viable way to make this stuff faster (parallel execution, fork once) it appears to me that the primary concurrency issue is due to how we initialize the test seed and the fact that we loop over all RandomWrapper objects and reset them. So, it's likely the case that in mid stream of some of the tests, the RNG is getting reset by other calls to the static useTestSeed() method.
>
> Of course, there might be other concurrency issues beyond that, but this seems like the most likely one to start. Thus, the question is how to fix it. The obvious one, I suppose, is to not use statics for this stuff. Another is, to perhaps, use a system property (-DuseTestSeed=true and/or -DuseSeed=<SEED>, the latter being useful for debugging other things) that is set upon invocation in the test plugin, but has the downside that it would also need to be set when running from an IDE.
>
> And, to Sean's point below, it seems that we may have some test dependencies on the specific set of random numbers and the outcomes they produce.
>
> Thoughts? Other ideas?
>
>
> On Dec 8, 2011, at 1:05 PM, Grant Ingersoll wrote:
>
>> Progress! I had configured the surefire plugin in the wrong place
>>
>>
>> On Dec 8, 2011, at 2:55 PM, Sean Owen wrote:
>>
>>> This could well be it. While every Random everywhere gets initialized to a
>>> known initial state, at the start of every @Test method, you could get
>>> different sequences if other tests are in progress in parallel in the same
>>> JVM.
>>>
>>> Ideally tests aren't that sensitive to the sequence of random numbers -- if
>>> that's the case. And here it may well be the case.
>>>
>>> Can this be set to fork a JVM per test class? that would probably work.
>>>
>>> On Thu, Dec 8, 2011 at 7:43 PM, Grant Ingersoll <gs...@apache.org> wrote:
>>>
>>>>
>>>> On Dec 8, 2011, at 2:39 PM, Grant Ingersoll wrote:
>>>>
>>>>>
>>>>> On Dec 8, 2011, at 2:23 PM, Grant Ingersoll wrote:
>>>>>
>>>>>> If I add parallel, fork always to the main surefire config, I get
>>>> failures all over the place for things like:
>>>>>> Failed tests:
>>>> testHebbianSolver(org.apache.mahout.math.decomposer.hebbian.TestHebbianSolver):
>>>> Error: {0.06146049974880152 too high! (for eigen 3)
>>>>>> consistency(org.apache.mahout.math.jet.random.NormalTest):
>>>> offset=0.000 scale=1.000 Z = 8.2
>>>>>> consistency(org.apache.mahout.math.jet.random.ExponentialTest):
>>>> offset=0.000 scale=100.000 Z = 8.7
>>>>>>
>>>>>
>>>>> Check that, it seems each run can produce different failures, which
>>>> leads me to believe we have some shared values in our tests
>>>>
>>>> Random.getRandom() the culprit, perhaps?
>>>>
>>>>>
>>>>>
>>>>>> All of these pass individually and when not in parallel for me.
>>>>>>
>>>>>> Here's my config:
>>>>>> <plugin>
>>>>>> <groupId>org.apache.maven.plugins</groupId>
>>>>>> <artifactId>maven-surefire-plugin</artifactId>
>>>>>> <version>2.11</version>
>>>>>> <configuration>
>>>>>> <parallel>classes</parallel>
>>>>>> <forkMode>always</forkMode>
>>>>>> <perCoreThreadCount>true</perCoreThreadCount>
>>>>>> </configuration>
>>>>>> </plugin>
>>>>>>
>>>>>> Anyone else seeing that?
>>>>>>
>>>>>>
>>>>>> On Dec 8, 2011, at 1:53 PM, Dmitriy Lyubimov wrote:
>>>>>>
>>>>>>> SSVD actually runs a rather small test but it is a MR job in local
>>>>>>> mode, there's nothing to cut down there in terms of size (not much
>>>>>>> anyway). It's just what it takes to initialize and run all jobs (and
>>>>>>> since it is local, it is also single threaded, so it actually runs V
>>>>>>> and U jobs sequentially instead of parallel so it's even longer
>>>>>>> because of that (4 jobs stringed all in all).
>>>>>>>
>>>>>>> But i will take a look, although even if i reduce solution size, it
>>>>>>> will still likely not reduce running time by more than 20%.
>>>>>>>
>>>>>>> On Thu, Dec 8, 2011 at 5:42 AM, David Murgatroyd <dm...@gmail.com>
>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Dec 8, 2011, at 8:36 AM, Grant Ingersoll <gs...@apache.org>
>>>> wrote:
>>>>>>>>
>>>>>>>>> MAHOUT-916 and 917 are attempts to address the running time of our
>>>> tests. As Sean rightfully pointed out, there are probably opportunities to
>>>> simply cut down the sizes of some of these tests w/o effecting there
>>>> correctness. To that end, if people can take a look at:
>>>>>>>>> https://builds.apache.org/job/Mahout-Quality/1237/testReport/junit/
>>>>>>>>>
>>>>>>>>> You can get a sense as to which tests are taking a long time. The
>>>> main culprits are:
>>>>>>>>> 1. Vectorizer
>>>>>>>>> 2. SSVD
>>>>>>>>> 3. K-Means
>>>>>>>>> 4. taste.hadoop.item
>>>>>>>>> 5. taste.hadoop.als
>>>>>>>>> 6. PFPGrowth
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -Grant
>>>>>>>>>
>>>>>>>>> --------------------------------------------
>>>>>>>>> Grant Ingersoll
>>>>>>>>> http://www.lucidimagination.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>> --------------------------------------------
>>>>>> Grant Ingersoll
>>>>>> http://www.lucidimagination.com
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --------------------------------------------
>>>>> Grant Ingersoll
>>>>> http://www.lucidimagination.com
>>>>>
>>>>>
>>>>>
>>>>
>>>> --------------------------------------------
>>>> Grant Ingersoll
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>>
>>>>
>>
>> --------------------------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com
>>
>>
>>
>
> --------------------------------------------
> Grant Ingersoll
> http://www.lucidimagination.com
>
>
>
--------------------------------------------
Grant Ingersoll
http://www.lucidimagination.com
Re: Tests running time
Posted by Grant Ingersoll <gs...@apache.org>.
In working through what I _think_ will be the primary viable way to make this stuff faster (parallel execution, fork once) it appears to me that the primary concurrency issue is due to how we initialize the test seed and the fact that we loop over all RandomWrapper objects and reset them. So, it's likely the case that in mid stream of some of the tests, the RNG is getting reset by other calls to the static useTestSeed() method.
Of course, there might be other concurrency issues beyond that, but this seems like the most likely one to start. Thus, the question is how to fix it. The obvious one, I suppose, is to not use statics for this stuff. Another is, to perhaps, use a system property (-DuseTestSeed=true and/or -DuseSeed=<SEED>, the latter being useful for debugging other things) that is set upon invocation in the test plugin, but has the downside that it would also need to be set when running from an IDE.
And, to Sean's point below, it seems that we may have some test dependencies on the specific set of random numbers and the outcomes they produce.
Thoughts? Other ideas?
On Dec 8, 2011, at 1:05 PM, Grant Ingersoll wrote:
> Progress! I had configured the surefire plugin in the wrong place
>
>
> On Dec 8, 2011, at 2:55 PM, Sean Owen wrote:
>
>> This could well be it. While every Random everywhere gets initialized to a
>> known initial state, at the start of every @Test method, you could get
>> different sequences if other tests are in progress in parallel in the same
>> JVM.
>>
>> Ideally tests aren't that sensitive to the sequence of random numbers -- if
>> that's the case. And here it may well be the case.
>>
>> Can this be set to fork a JVM per test class? that would probably work.
>>
>> On Thu, Dec 8, 2011 at 7:43 PM, Grant Ingersoll <gs...@apache.org> wrote:
>>
>>>
>>> On Dec 8, 2011, at 2:39 PM, Grant Ingersoll wrote:
>>>
>>>>
>>>> On Dec 8, 2011, at 2:23 PM, Grant Ingersoll wrote:
>>>>
>>>>> If I add parallel, fork always to the main surefire config, I get
>>> failures all over the place for things like:
>>>>> Failed tests:
>>> testHebbianSolver(org.apache.mahout.math.decomposer.hebbian.TestHebbianSolver):
>>> Error: {0.06146049974880152 too high! (for eigen 3)
>>>>> consistency(org.apache.mahout.math.jet.random.NormalTest):
>>> offset=0.000 scale=1.000 Z = 8.2
>>>>> consistency(org.apache.mahout.math.jet.random.ExponentialTest):
>>> offset=0.000 scale=100.000 Z = 8.7
>>>>>
>>>>
>>>> Check that, it seems each run can produce different failures, which
>>> leads me to believe we have some shared values in our tests
>>>
>>> Random.getRandom() the culprit, perhaps?
>>>
>>>>
>>>>
>>>>> All of these pass individually and when not in parallel for me.
>>>>>
>>>>> Here's my config:
>>>>> <plugin>
>>>>> <groupId>org.apache.maven.plugins</groupId>
>>>>> <artifactId>maven-surefire-plugin</artifactId>
>>>>> <version>2.11</version>
>>>>> <configuration>
>>>>> <parallel>classes</parallel>
>>>>> <forkMode>always</forkMode>
>>>>> <perCoreThreadCount>true</perCoreThreadCount>
>>>>> </configuration>
>>>>> </plugin>
>>>>>
>>>>> Anyone else seeing that?
>>>>>
>>>>>
>>>>> On Dec 8, 2011, at 1:53 PM, Dmitriy Lyubimov wrote:
>>>>>
>>>>>> SSVD actually runs a rather small test but it is a MR job in local
>>>>>> mode, there's nothing to cut down there in terms of size (not much
>>>>>> anyway). It's just what it takes to initialize and run all jobs (and
>>>>>> since it is local, it is also single threaded, so it actually runs V
>>>>>> and U jobs sequentially instead of parallel so it's even longer
>>>>>> because of that (4 jobs stringed all in all).
>>>>>>
>>>>>> But i will take a look, although even if i reduce solution size, it
>>>>>> will still likely not reduce running time by more than 20%.
>>>>>>
>>>>>> On Thu, Dec 8, 2011 at 5:42 AM, David Murgatroyd <dm...@gmail.com>
>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Dec 8, 2011, at 8:36 AM, Grant Ingersoll <gs...@apache.org>
>>> wrote:
>>>>>>>
>>>>>>>> MAHOUT-916 and 917 are attempts to address the running time of our
>>> tests. As Sean rightfully pointed out, there are probably opportunities to
>>> simply cut down the sizes of some of these tests w/o effecting there
>>> correctness. To that end, if people can take a look at:
>>>>>>>> https://builds.apache.org/job/Mahout-Quality/1237/testReport/junit/
>>>>>>>>
>>>>>>>> You can get a sense as to which tests are taking a long time. The
>>> main culprits are:
>>>>>>>> 1. Vectorizer
>>>>>>>> 2. SSVD
>>>>>>>> 3. K-Means
>>>>>>>> 4. taste.hadoop.item
>>>>>>>> 5. taste.hadoop.als
>>>>>>>> 6. PFPGrowth
>>>>>>>>
>>>>>>>>
>>>>>>>> -Grant
>>>>>>>>
>>>>>>>> --------------------------------------------
>>>>>>>> Grant Ingersoll
>>>>>>>> http://www.lucidimagination.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>
>>>>> --------------------------------------------
>>>>> Grant Ingersoll
>>>>> http://www.lucidimagination.com
>>>>>
>>>>>
>>>>>
>>>>
>>>> --------------------------------------------
>>>> Grant Ingersoll
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>>
>>>
>>> --------------------------------------------
>>> Grant Ingersoll
>>> http://www.lucidimagination.com
>>>
>>>
>>>
>>>
>
> --------------------------------------------
> Grant Ingersoll
> http://www.lucidimagination.com
>
>
>
--------------------------------------------
Grant Ingersoll
http://www.lucidimagination.com
Re: Tests running time
Posted by Grant Ingersoll <gs...@apache.org>.
Progress! I had configured the surefire plugin in the wrong place
On Dec 8, 2011, at 2:55 PM, Sean Owen wrote:
> This could well be it. While every Random everywhere gets initialized to a
> known initial state, at the start of every @Test method, you could get
> different sequences if other tests are in progress in parallel in the same
> JVM.
>
> Ideally tests aren't that sensitive to the sequence of random numbers -- if
> that's the case. And here it may well be the case.
>
> Can this be set to fork a JVM per test class? that would probably work.
>
> On Thu, Dec 8, 2011 at 7:43 PM, Grant Ingersoll <gs...@apache.org> wrote:
>
>>
>> On Dec 8, 2011, at 2:39 PM, Grant Ingersoll wrote:
>>
>>>
>>> On Dec 8, 2011, at 2:23 PM, Grant Ingersoll wrote:
>>>
>>>> If I add parallel, fork always to the main surefire config, I get
>> failures all over the place for things like:
>>>> Failed tests:
>> testHebbianSolver(org.apache.mahout.math.decomposer.hebbian.TestHebbianSolver):
>> Error: {0.06146049974880152 too high! (for eigen 3)
>>>> consistency(org.apache.mahout.math.jet.random.NormalTest):
>> offset=0.000 scale=1.000 Z = 8.2
>>>> consistency(org.apache.mahout.math.jet.random.ExponentialTest):
>> offset=0.000 scale=100.000 Z = 8.7
>>>>
>>>
>>> Check that, it seems each run can produce different failures, which
>> leads me to believe we have some shared values in our tests
>>
>> Random.getRandom() the culprit, perhaps?
>>
>>>
>>>
>>>> All of these pass individually and when not in parallel for me.
>>>>
>>>> Here's my config:
>>>> <plugin>
>>>> <groupId>org.apache.maven.plugins</groupId>
>>>> <artifactId>maven-surefire-plugin</artifactId>
>>>> <version>2.11</version>
>>>> <configuration>
>>>> <parallel>classes</parallel>
>>>> <forkMode>always</forkMode>
>>>> <perCoreThreadCount>true</perCoreThreadCount>
>>>> </configuration>
>>>> </plugin>
>>>>
>>>> Anyone else seeing that?
>>>>
>>>>
>>>> On Dec 8, 2011, at 1:53 PM, Dmitriy Lyubimov wrote:
>>>>
>>>>> SSVD actually runs a rather small test but it is a MR job in local
>>>>> mode, there's nothing to cut down there in terms of size (not much
>>>>> anyway). It's just what it takes to initialize and run all jobs (and
>>>>> since it is local, it is also single threaded, so it actually runs V
>>>>> and U jobs sequentially instead of parallel so it's even longer
>>>>> because of that (4 jobs stringed all in all).
>>>>>
>>>>> But i will take a look, although even if i reduce solution size, it
>>>>> will still likely not reduce running time by more than 20%.
>>>>>
>>>>> On Thu, Dec 8, 2011 at 5:42 AM, David Murgatroyd <dm...@gmail.com>
>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Dec 8, 2011, at 8:36 AM, Grant Ingersoll <gs...@apache.org>
>> wrote:
>>>>>>
>>>>>>> MAHOUT-916 and 917 are attempts to address the running time of our
>> tests. As Sean rightfully pointed out, there are probably opportunities to
>> simply cut down the sizes of some of these tests w/o effecting there
>> correctness. To that end, if people can take a look at:
>>>>>>> https://builds.apache.org/job/Mahout-Quality/1237/testReport/junit/
>>>>>>>
>>>>>>> You can get a sense as to which tests are taking a long time. The
>> main culprits are:
>>>>>>> 1. Vectorizer
>>>>>>> 2. SSVD
>>>>>>> 3. K-Means
>>>>>>> 4. taste.hadoop.item
>>>>>>> 5. taste.hadoop.als
>>>>>>> 6. PFPGrowth
>>>>>>>
>>>>>>>
>>>>>>> -Grant
>>>>>>>
>>>>>>> --------------------------------------------
>>>>>>> Grant Ingersoll
>>>>>>> http://www.lucidimagination.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>
>>>> --------------------------------------------
>>>> Grant Ingersoll
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>>
>>>
>>> --------------------------------------------
>>> Grant Ingersoll
>>> http://www.lucidimagination.com
>>>
>>>
>>>
>>
>> --------------------------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com
>>
>>
>>
>>
--------------------------------------------
Grant Ingersoll
http://www.lucidimagination.com
Re: Tests running time
Posted by Sean Owen <sr...@gmail.com>.
I think that just means the test is dependent on the particular sequence of
random numbers. Ideally it should loosen its definition of correct a bit.
On Thu, Dec 8, 2011 at 7:59 PM, Grant Ingersoll <gs...@apache.org> wrote:
>
> On Dec 8, 2011, at 2:55 PM, Sean Owen wrote:
>
> > This could well be it. While every Random everywhere gets initialized to
> a
> > known initial state, at the start of every @Test method, you could get
> > different sequences if other tests are in progress in parallel in the
> same
> > JVM.
>
> I'm also trying using the test seed, which passes sequentially, but still
> fails
>
>
Re: Tests running time
Posted by Grant Ingersoll <gs...@apache.org>.
On Dec 8, 2011, at 2:55 PM, Sean Owen wrote:
> This could well be it. While every Random everywhere gets initialized to a
> known initial state, at the start of every @Test method, you could get
> different sequences if other tests are in progress in parallel in the same
> JVM.
I'm also trying using the test seed, which passes sequentially, but still fails
>
> Ideally tests aren't that sensitive to the sequence of random numbers -- if
> that's the case. And here it may well be the case.
>
> Can this be set to fork a JVM per test class? that would probably work.
Will try.
>
> On Thu, Dec 8, 2011 at 7:43 PM, Grant Ingersoll <gs...@apache.org> wrote:
>
>>
>> On Dec 8, 2011, at 2:39 PM, Grant Ingersoll wrote:
>>
>>>
>>> On Dec 8, 2011, at 2:23 PM, Grant Ingersoll wrote:
>>>
>>>> If I add parallel, fork always to the main surefire config, I get
>> failures all over the place for things like:
>>>> Failed tests:
>> testHebbianSolver(org.apache.mahout.math.decomposer.hebbian.TestHebbianSolver):
>> Error: {0.06146049974880152 too high! (for eigen 3)
>>>> consistency(org.apache.mahout.math.jet.random.NormalTest):
>> offset=0.000 scale=1.000 Z = 8.2
>>>> consistency(org.apache.mahout.math.jet.random.ExponentialTest):
>> offset=0.000 scale=100.000 Z = 8.7
>>>>
>>>
>>> Check that, it seems each run can produce different failures, which
>> leads me to believe we have some shared values in our tests
>>
>> Random.getRandom() the culprit, perhaps?
>>
>>>
>>>
>>>> All of these pass individually and when not in parallel for me.
>>>>
>>>> Here's my config:
>>>> <plugin>
>>>> <groupId>org.apache.maven.plugins</groupId>
>>>> <artifactId>maven-surefire-plugin</artifactId>
>>>> <version>2.11</version>
>>>> <configuration>
>>>> <parallel>classes</parallel>
>>>> <forkMode>always</forkMode>
>>>> <perCoreThreadCount>true</perCoreThreadCount>
>>>> </configuration>
>>>> </plugin>
>>>>
>>>> Anyone else seeing that?
>>>>
>>>>
>>>> On Dec 8, 2011, at 1:53 PM, Dmitriy Lyubimov wrote:
>>>>
>>>>> SSVD actually runs a rather small test but it is a MR job in local
>>>>> mode, there's nothing to cut down there in terms of size (not much
>>>>> anyway). It's just what it takes to initialize and run all jobs (and
>>>>> since it is local, it is also single threaded, so it actually runs V
>>>>> and U jobs sequentially instead of parallel so it's even longer
>>>>> because of that (4 jobs stringed all in all).
>>>>>
>>>>> But i will take a look, although even if i reduce solution size, it
>>>>> will still likely not reduce running time by more than 20%.
>>>>>
>>>>> On Thu, Dec 8, 2011 at 5:42 AM, David Murgatroyd <dm...@gmail.com>
>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Dec 8, 2011, at 8:36 AM, Grant Ingersoll <gs...@apache.org>
>> wrote:
>>>>>>
>>>>>>> MAHOUT-916 and 917 are attempts to address the running time of our
>> tests. As Sean rightfully pointed out, there are probably opportunities to
>> simply cut down the sizes of some of these tests w/o effecting there
>> correctness. To that end, if people can take a look at:
>>>>>>> https://builds.apache.org/job/Mahout-Quality/1237/testReport/junit/
>>>>>>>
>>>>>>> You can get a sense as to which tests are taking a long time. The
>> main culprits are:
>>>>>>> 1. Vectorizer
>>>>>>> 2. SSVD
>>>>>>> 3. K-Means
>>>>>>> 4. taste.hadoop.item
>>>>>>> 5. taste.hadoop.als
>>>>>>> 6. PFPGrowth
>>>>>>>
>>>>>>>
>>>>>>> -Grant
>>>>>>>
>>>>>>> --------------------------------------------
>>>>>>> Grant Ingersoll
>>>>>>> http://www.lucidimagination.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>
>>>> --------------------------------------------
>>>> Grant Ingersoll
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>>
>>>
>>> --------------------------------------------
>>> Grant Ingersoll
>>> http://www.lucidimagination.com
>>>
>>>
>>>
>>
>> --------------------------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com
>>
>>
>>
>>
--------------------------------------------
Grant Ingersoll
http://www.lucidimagination.com
Re: Tests running time
Posted by Sean Owen <sr...@gmail.com>.
This could well be it. While every Random everywhere gets initialized to a
known initial state, at the start of every @Test method, you could get
different sequences if other tests are in progress in parallel in the same
JVM.
Ideally tests aren't that sensitive to the sequence of random numbers -- if
that's the case. And here it may well be the case.
Can this be set to fork a JVM per test class? that would probably work.
On Thu, Dec 8, 2011 at 7:43 PM, Grant Ingersoll <gs...@apache.org> wrote:
>
> On Dec 8, 2011, at 2:39 PM, Grant Ingersoll wrote:
>
> >
> > On Dec 8, 2011, at 2:23 PM, Grant Ingersoll wrote:
> >
> >> If I add parallel, fork always to the main surefire config, I get
> failures all over the place for things like:
> >> Failed tests:
> testHebbianSolver(org.apache.mahout.math.decomposer.hebbian.TestHebbianSolver):
> Error: {0.06146049974880152 too high! (for eigen 3)
> >> consistency(org.apache.mahout.math.jet.random.NormalTest):
> offset=0.000 scale=1.000 Z = 8.2
> >> consistency(org.apache.mahout.math.jet.random.ExponentialTest):
> offset=0.000 scale=100.000 Z = 8.7
> >>
> >
> > Check that, it seems each run can produce different failures, which
> leads me to believe we have some shared values in our tests
>
> Random.getRandom() the culprit, perhaps?
>
> >
> >
> >> All of these pass individually and when not in parallel for me.
> >>
> >> Here's my config:
> >> <plugin>
> >> <groupId>org.apache.maven.plugins</groupId>
> >> <artifactId>maven-surefire-plugin</artifactId>
> >> <version>2.11</version>
> >> <configuration>
> >> <parallel>classes</parallel>
> >> <forkMode>always</forkMode>
> >> <perCoreThreadCount>true</perCoreThreadCount>
> >> </configuration>
> >> </plugin>
> >>
> >> Anyone else seeing that?
> >>
> >>
> >> On Dec 8, 2011, at 1:53 PM, Dmitriy Lyubimov wrote:
> >>
> >>> SSVD actually runs a rather small test but it is a MR job in local
> >>> mode, there's nothing to cut down there in terms of size (not much
> >>> anyway). It's just what it takes to initialize and run all jobs (and
> >>> since it is local, it is also single threaded, so it actually runs V
> >>> and U jobs sequentially instead of parallel so it's even longer
> >>> because of that (4 jobs stringed all in all).
> >>>
> >>> But i will take a look, although even if i reduce solution size, it
> >>> will still likely not reduce running time by more than 20%.
> >>>
> >>> On Thu, Dec 8, 2011 at 5:42 AM, David Murgatroyd <dm...@gmail.com>
> wrote:
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Dec 8, 2011, at 8:36 AM, Grant Ingersoll <gs...@apache.org>
> wrote:
> >>>>
> >>>>> MAHOUT-916 and 917 are attempts to address the running time of our
> tests. As Sean rightfully pointed out, there are probably opportunities to
> simply cut down the sizes of some of these tests w/o effecting there
> correctness. To that end, if people can take a look at:
> >>>>> https://builds.apache.org/job/Mahout-Quality/1237/testReport/junit/
> >>>>>
> >>>>> You can get a sense as to which tests are taking a long time. The
> main culprits are:
> >>>>> 1. Vectorizer
> >>>>> 2. SSVD
> >>>>> 3. K-Means
> >>>>> 4. taste.hadoop.item
> >>>>> 5. taste.hadoop.als
> >>>>> 6. PFPGrowth
> >>>>>
> >>>>>
> >>>>> -Grant
> >>>>>
> >>>>> --------------------------------------------
> >>>>> Grant Ingersoll
> >>>>> http://www.lucidimagination.com
> >>>>>
> >>>>>
> >>>>>
> >>
> >> --------------------------------------------
> >> Grant Ingersoll
> >> http://www.lucidimagination.com
> >>
> >>
> >>
> >
> > --------------------------------------------
> > Grant Ingersoll
> > http://www.lucidimagination.com
> >
> >
> >
>
> --------------------------------------------
> Grant Ingersoll
> http://www.lucidimagination.com
>
>
>
>
Re: Tests running time
Posted by Grant Ingersoll <gs...@apache.org>.
On Dec 8, 2011, at 2:39 PM, Grant Ingersoll wrote:
>
> On Dec 8, 2011, at 2:23 PM, Grant Ingersoll wrote:
>
>> If I add parallel, fork always to the main surefire config, I get failures all over the place for things like:
>> Failed tests: testHebbianSolver(org.apache.mahout.math.decomposer.hebbian.TestHebbianSolver): Error: {0.06146049974880152 too high! (for eigen 3)
>> consistency(org.apache.mahout.math.jet.random.NormalTest): offset=0.000 scale=1.000 Z = 8.2
>> consistency(org.apache.mahout.math.jet.random.ExponentialTest): offset=0.000 scale=100.000 Z = 8.7
>>
>
> Check that, it seems each run can produce different failures, which leads me to believe we have some shared values in our tests
Random.getRandom() the culprit, perhaps?
>
>
>> All of these pass individually and when not in parallel for me.
>>
>> Here's my config:
>> <plugin>
>> <groupId>org.apache.maven.plugins</groupId>
>> <artifactId>maven-surefire-plugin</artifactId>
>> <version>2.11</version>
>> <configuration>
>> <parallel>classes</parallel>
>> <forkMode>always</forkMode>
>> <perCoreThreadCount>true</perCoreThreadCount>
>> </configuration>
>> </plugin>
>>
>> Anyone else seeing that?
>>
>>
>> On Dec 8, 2011, at 1:53 PM, Dmitriy Lyubimov wrote:
>>
>>> SSVD actually runs a rather small test but it is a MR job in local
>>> mode, there's nothing to cut down there in terms of size (not much
>>> anyway). It's just what it takes to initialize and run all jobs (and
>>> since it is local, it is also single threaded, so it actually runs V
>>> and U jobs sequentially instead of parallel so it's even longer
>>> because of that (4 jobs stringed all in all).
>>>
>>> But i will take a look, although even if i reduce solution size, it
>>> will still likely not reduce running time by more than 20%.
>>>
>>> On Thu, Dec 8, 2011 at 5:42 AM, David Murgatroyd <dm...@gmail.com> wrote:
>>>>
>>>>
>>>>
>>>>
>>>> On Dec 8, 2011, at 8:36 AM, Grant Ingersoll <gs...@apache.org> wrote:
>>>>
>>>>> MAHOUT-916 and 917 are attempts to address the running time of our tests. As Sean rightfully pointed out, there are probably opportunities to simply cut down the sizes of some of these tests w/o effecting there correctness. To that end, if people can take a look at:
>>>>> https://builds.apache.org/job/Mahout-Quality/1237/testReport/junit/
>>>>>
>>>>> You can get a sense as to which tests are taking a long time. The main culprits are:
>>>>> 1. Vectorizer
>>>>> 2. SSVD
>>>>> 3. K-Means
>>>>> 4. taste.hadoop.item
>>>>> 5. taste.hadoop.als
>>>>> 6. PFPGrowth
>>>>>
>>>>>
>>>>> -Grant
>>>>>
>>>>> --------------------------------------------
>>>>> Grant Ingersoll
>>>>> http://www.lucidimagination.com
>>>>>
>>>>>
>>>>>
>>
>> --------------------------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com
>>
>>
>>
>
> --------------------------------------------
> Grant Ingersoll
> http://www.lucidimagination.com
>
>
>
--------------------------------------------
Grant Ingersoll
http://www.lucidimagination.com
Re: Tests running time
Posted by Grant Ingersoll <gs...@apache.org>.
On Dec 8, 2011, at 2:23 PM, Grant Ingersoll wrote:
> If I add parallel, fork always to the main surefire config, I get failures all over the place for things like:
> Failed tests: testHebbianSolver(org.apache.mahout.math.decomposer.hebbian.TestHebbianSolver): Error: {0.06146049974880152 too high! (for eigen 3)
> consistency(org.apache.mahout.math.jet.random.NormalTest): offset=0.000 scale=1.000 Z = 8.2
> consistency(org.apache.mahout.math.jet.random.ExponentialTest): offset=0.000 scale=100.000 Z = 8.7
>
Check that, it seems each run can produce different failures, which leads me to believe we have some shared values in our tests
> All of these pass individually and when not in parallel for me.
>
> Here's my config:
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> <version>2.11</version>
> <configuration>
> <parallel>classes</parallel>
> <forkMode>always</forkMode>
> <perCoreThreadCount>true</perCoreThreadCount>
> </configuration>
> </plugin>
>
> Anyone else seeing that?
>
>
> On Dec 8, 2011, at 1:53 PM, Dmitriy Lyubimov wrote:
>
>> SSVD actually runs a rather small test but it is a MR job in local
>> mode, there's nothing to cut down there in terms of size (not much
>> anyway). It's just what it takes to initialize and run all jobs (and
>> since it is local, it is also single threaded, so it actually runs V
>> and U jobs sequentially instead of parallel so it's even longer
>> because of that (4 jobs stringed all in all).
>>
>> But i will take a look, although even if i reduce solution size, it
>> will still likely not reduce running time by more than 20%.
>>
>> On Thu, Dec 8, 2011 at 5:42 AM, David Murgatroyd <dm...@gmail.com> wrote:
>>>
>>>
>>>
>>>
>>> On Dec 8, 2011, at 8:36 AM, Grant Ingersoll <gs...@apache.org> wrote:
>>>
>>>> MAHOUT-916 and 917 are attempts to address the running time of our tests. As Sean rightfully pointed out, there are probably opportunities to simply cut down the sizes of some of these tests w/o effecting there correctness. To that end, if people can take a look at:
>>>> https://builds.apache.org/job/Mahout-Quality/1237/testReport/junit/
>>>>
>>>> You can get a sense as to which tests are taking a long time. The main culprits are:
>>>> 1. Vectorizer
>>>> 2. SSVD
>>>> 3. K-Means
>>>> 4. taste.hadoop.item
>>>> 5. taste.hadoop.als
>>>> 6. PFPGrowth
>>>>
>>>>
>>>> -Grant
>>>>
>>>> --------------------------------------------
>>>> Grant Ingersoll
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>>
>
> --------------------------------------------
> Grant Ingersoll
> http://www.lucidimagination.com
>
>
>
--------------------------------------------
Grant Ingersoll
http://www.lucidimagination.com
Re: Tests running time
Posted by Grant Ingersoll <gs...@apache.org>.
If I add parallel, fork always to the main surefire config, I get failures all over the place for things like:
Failed tests: testHebbianSolver(org.apache.mahout.math.decomposer.hebbian.TestHebbianSolver): Error: {0.06146049974880152 too high! (for eigen 3)
consistency(org.apache.mahout.math.jet.random.NormalTest): offset=0.000 scale=1.000 Z = 8.2
consistency(org.apache.mahout.math.jet.random.ExponentialTest): offset=0.000 scale=100.000 Z = 8.7
All of these pass individually and when not in parallel for me.
Here's my config:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.11</version>
<configuration>
<parallel>classes</parallel>
<forkMode>always</forkMode>
<perCoreThreadCount>true</perCoreThreadCount>
</configuration>
</plugin>
Anyone else seeing that?
On Dec 8, 2011, at 1:53 PM, Dmitriy Lyubimov wrote:
> SSVD actually runs a rather small test but it is a MR job in local
> mode, there's nothing to cut down there in terms of size (not much
> anyway). It's just what it takes to initialize and run all jobs (and
> since it is local, it is also single threaded, so it actually runs V
> and U jobs sequentially instead of parallel so it's even longer
> because of that (4 jobs stringed all in all).
>
> But i will take a look, although even if i reduce solution size, it
> will still likely not reduce running time by more than 20%.
>
> On Thu, Dec 8, 2011 at 5:42 AM, David Murgatroyd <dm...@gmail.com> wrote:
>>
>>
>>
>>
>> On Dec 8, 2011, at 8:36 AM, Grant Ingersoll <gs...@apache.org> wrote:
>>
>>> MAHOUT-916 and 917 are attempts to address the running time of our tests. As Sean rightfully pointed out, there are probably opportunities to simply cut down the sizes of some of these tests w/o effecting there correctness. To that end, if people can take a look at:
>>> https://builds.apache.org/job/Mahout-Quality/1237/testReport/junit/
>>>
>>> You can get a sense as to which tests are taking a long time. The main culprits are:
>>> 1. Vectorizer
>>> 2. SSVD
>>> 3. K-Means
>>> 4. taste.hadoop.item
>>> 5. taste.hadoop.als
>>> 6. PFPGrowth
>>>
>>>
>>> -Grant
>>>
>>> --------------------------------------------
>>> Grant Ingersoll
>>> http://www.lucidimagination.com
>>>
>>>
>>>
--------------------------------------------
Grant Ingersoll
http://www.lucidimagination.com
Re: Tests running time
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
SSVD actually runs a rather small test but it is a MR job in local
mode, there's nothing to cut down there in terms of size (not much
anyway). It's just what it takes to initialize and run all jobs (and
since it is local, it is also single threaded, so it actually runs V
and U jobs sequentially instead of parallel so it's even longer
because of that (4 jobs stringed all in all).
But i will take a look, although even if i reduce solution size, it
will still likely not reduce running time by more than 20%.
On Thu, Dec 8, 2011 at 5:42 AM, David Murgatroyd <dm...@gmail.com> wrote:
>
>
>
>
> On Dec 8, 2011, at 8:36 AM, Grant Ingersoll <gs...@apache.org> wrote:
>
>> MAHOUT-916 and 917 are attempts to address the running time of our tests. As Sean rightfully pointed out, there are probably opportunities to simply cut down the sizes of some of these tests w/o effecting there correctness. To that end, if people can take a look at:
>> https://builds.apache.org/job/Mahout-Quality/1237/testReport/junit/
>>
>> You can get a sense as to which tests are taking a long time. The main culprits are:
>> 1. Vectorizer
>> 2. SSVD
>> 3. K-Means
>> 4. taste.hadoop.item
>> 5. taste.hadoop.als
>> 6. PFPGrowth
>>
>>
>> -Grant
>>
>> --------------------------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com
>>
>>
>>
Re: Tests running time
Posted by David Murgatroyd <dm...@gmail.com>.
On Dec 8, 2011, at 8:36 AM, Grant Ingersoll <gs...@apache.org> wrote:
> MAHOUT-916 and 917 are attempts to address the running time of our tests. As Sean rightfully pointed out, there are probably opportunities to simply cut down the sizes of some of these tests w/o effecting there correctness. To that end, if people can take a look at:
> https://builds.apache.org/job/Mahout-Quality/1237/testReport/junit/
>
> You can get a sense as to which tests are taking a long time. The main culprits are:
> 1. Vectorizer
> 2. SSVD
> 3. K-Means
> 4. taste.hadoop.item
> 5. taste.hadoop.als
> 6. PFPGrowth
>
>
> -Grant
>
> --------------------------------------------
> Grant Ingersoll
> http://www.lucidimagination.com
>
>
>
Re: Tests running time
Posted by Lance Norskog <go...@gmail.com>.
The Map/Reduce SGD patch includes a very nice trick which I did not
know about. Here is an example:
https://reviews.apache.org/r/3072/diff/2/?file=63195#file63195line36
It uses DummyRecordWriter to ship key/value pairs from mapper to reducer.
On Mon, Dec 12, 2011 at 12:43 PM, Isabel Drost <is...@apache.org> wrote:
> On 08.12.2011 Grant Ingersoll wrote:
>> great, b/c these really are mainstream tests. I suspect most of our
>> overhead is simply due to running map reduce jobs.
>
> Is there anything to be gained by checking the code itself with mrunit (I know
> it does have limitations, but if those tests really only test map and reduce
> functions, it might be good enough) and moving integration tests to a "run only
> nightly" mode?
>
>
> Isabel
--
Lance Norskog
goksron@gmail.com
Re: Tests running time
Posted by Isabel Drost <is...@apache.org>.
On 08.12.2011 Grant Ingersoll wrote:
> great, b/c these really are mainstream tests. I suspect most of our
> overhead is simply due to running map reduce jobs.
Is there anything to be gained by checking the code itself with mrunit (I know
it does have limitations, but if those tests really only test map and reduce
functions, it might be good enough) and moving integration tests to a "run only
nightly" mode?
Isabel
Re: Tests running time
Posted by Grant Ingersoll <gs...@apache.org>.
I working on parallel at the moment, which should help if we can get it to work. We could certainly setup nightly, but I don't really think that is great, b/c these really are mainstream tests. I suspect most of our overhead is simply due to running map reduce jobs.
On Dec 8, 2011, at 8:42 AM, Sebastian Schelter wrote:
> 4. and 5. already run on toy data.
>
> I have some rather excessive 'integration'-like tests that execute
> Hadoop in a local JVM. The tests take very long but are also very
> helpful in finding subtle bugs.
>
> Maybe there is a way to execute these tests only once a day or so?
>
> --sebastian
>
> On 08.12.2011 14:36, Grant Ingersoll wrote:
>> MAHOUT-916 and 917 are attempts to address the running time of our tests. As Sean rightfully pointed out, there are probably opportunities to simply cut down the sizes of some of these tests w/o effecting there correctness. To that end, if people can take a look at:
>> https://builds.apache.org/job/Mahout-Quality/1237/testReport/junit/
>>
>> You can get a sense as to which tests are taking a long time. The main culprits are:
>> 1. Vectorizer
>> 2. SSVD
>> 3. K-Means
>> 4. taste.hadoop.item
>> 5. taste.hadoop.als
>> 6. PFPGrowth
>>
>>
>> -Grant
>>
>> --------------------------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com
>>
>>
>>
>>
>
Re: Tests running time
Posted by Sebastian Schelter <ss...@apache.org>.
4. and 5. already run on toy data.
I have some rather excessive 'integration'-like tests that execute
Hadoop in a local JVM. The tests take very long but are also very
helpful in finding subtle bugs.
Maybe there is a way to execute these tests only once a day or so?
--sebastian
On 08.12.2011 14:36, Grant Ingersoll wrote:
> MAHOUT-916 and 917 are attempts to address the running time of our tests. As Sean rightfully pointed out, there are probably opportunities to simply cut down the sizes of some of these tests w/o effecting there correctness. To that end, if people can take a look at:
> https://builds.apache.org/job/Mahout-Quality/1237/testReport/junit/
>
> You can get a sense as to which tests are taking a long time. The main culprits are:
> 1. Vectorizer
> 2. SSVD
> 3. K-Means
> 4. taste.hadoop.item
> 5. taste.hadoop.als
> 6. PFPGrowth
>
>
> -Grant
>
> --------------------------------------------
> Grant Ingersoll
> http://www.lucidimagination.com
>
>
>
>