You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Benson Margulies (JIRA)" <ji...@apache.org> on 2010/01/18 04:24:54 UTC
[jira] Created: (MAHOUT-260) An alternative approach to RNG
management
An alternative approach to RNG management
-----------------------------------------
Key: MAHOUT-260
URL: https://issues.apache.org/jira/browse/MAHOUT-260
Project: Mahout
Issue Type: Improvement
Affects Versions: 0.3
Reporter: Benson Margulies
Here's code to (a) use the JDK for random numbers when testing, and (b) make the RNG selectable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (MAHOUT-260) An alternative approach to RNG
management
Posted by Ted Dunning <te...@gmail.com>.
This just avoids the class load in the test. I don't think it is necessary.
On Mon, Jan 18, 2010 at 1:04 AM, Sean Owen (JIRA) <ji...@apache.org> wrote:
> I still don't understand what this solves. We already 'fixed' the
> performance issue.
>
--
Ted Dunning, CTO
DeepDyve
Re: [jira] Commented: (MAHOUT-260) An alternative approach to RNG
management
Posted by Ted Dunning <te...@gmail.com>.
I vote for fixing theoretical problems in a theoretical time frame.
I will be writing all code from now to allow an constructor injectable
random generator, but understand how the transitive injection could get to
be a pain for Sean in the Taste stuff.
On Thu, Jan 21, 2010 at 12:01 PM, Sean Owen (JIRA) <ji...@apache.org> wrote:
> But, it's a problem we have only in theory rather than practice right now.
>
--
Ted Dunning, CTO
DeepDyve
[jira] Updated: (MAHOUT-260) An alternative approach to RNG
management
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated MAHOUT-260:
-----------------------------
Priority: Minor (was: Major)
Fix Version/s: (was: 0.3)
Assignee: Sean Owen
I'd like to revisit this -- well, the issue I'm trying to attack with my patch.
I'm writing a lot more unit tests for my stuff and am hitting the repeatability problem above. It comes up pretty fast when directly testing random-related classes like SamplingLongPrimitiveIterator.
My patch does indeed fix it. It looks a little heavy, but is the correct way to do it.
This isn't a burning issue, though I'd like to commit my tests eventually. But wanted to reopen for direct discussion of the patch.
> An alternative approach to RNG management
> -----------------------------------------
>
> Key: MAHOUT-260
> URL: https://issues.apache.org/jira/browse/MAHOUT-260
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.3
> Reporter: Benson Margulies
> Assignee: Sean Owen
> Priority: Minor
> Attachments: MAHOUT-260.diff, MAHOUT-260_wrapper.patch
>
>
> Here's code to (a) use the JDK for random numbers when testing, and (b) make the RNG selectable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-260) An alternative approach to RNG
management
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801708#action_12801708 ]
Sean Owen commented on MAHOUT-260:
----------------------------------
I still don't understand what this solves. We already 'fixed' the performance issue.
This does not address the actual repeatability issue we've been discussing either.
This also breaks tests by changing the RNG that the tests are written to expect.
> An alternative approach to RNG management
> -----------------------------------------
>
> Key: MAHOUT-260
> URL: https://issues.apache.org/jira/browse/MAHOUT-260
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.3
> Reporter: Benson Margulies
> Attachments: MAHOUT-260.diff
>
>
> Here's code to (a) use the JDK for random numbers when testing, and (b) make the RNG selectable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-260) An alternative approach to RNG
management
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated MAHOUT-260:
-----------------------------
Attachment: MAHOUT-260_wrapper.patch
This is the wrapper-related change I had in mind. I think this really gets at the heart of the problem. It's not entirely pretty.
> An alternative approach to RNG management
> -----------------------------------------
>
> Key: MAHOUT-260
> URL: https://issues.apache.org/jira/browse/MAHOUT-260
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.3
> Reporter: Benson Margulies
> Attachments: MAHOUT-260.diff, MAHOUT-260_wrapper.patch
>
>
> Here's code to (a) use the JDK for random numbers when testing, and (b) make the RNG selectable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-260) An alternative approach to RNG
management
Posted by "Ted Dunning (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12834987#action_12834987 ]
Ted Dunning commented on MAHOUT-260:
------------------------------------
Sean,
Why did you use a map for storing the instances? Why not a set?
Otherwise, this code seems fine.
> An alternative approach to RNG management
> -----------------------------------------
>
> Key: MAHOUT-260
> URL: https://issues.apache.org/jira/browse/MAHOUT-260
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.3
> Reporter: Benson Margulies
> Assignee: Sean Owen
> Priority: Minor
> Attachments: MAHOUT-260.diff, MAHOUT-260_wrapper.patch
>
>
> Here's code to (a) use the JDK for random numbers when testing, and (b) make the RNG selectable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-260) An alternative approach to RNG
management
Posted by "Ted Dunning (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12835088#action_12835088 ]
Ted Dunning commented on MAHOUT-260:
------------------------------------
{quote}
WeakHashMap is the best the JDK offers short of rolling our own.
{quote}
You just did.
I wish google would hurry up and mavenize guava so we can all start using collections (and guava).
> An alternative approach to RNG management
> -----------------------------------------
>
> Key: MAHOUT-260
> URL: https://issues.apache.org/jira/browse/MAHOUT-260
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.3
> Reporter: Benson Margulies
> Assignee: Sean Owen
> Priority: Minor
> Attachments: MAHOUT-260.diff, MAHOUT-260_wrapper.patch
>
>
> Here's code to (a) use the JDK for random numbers when testing, and (b) make the RNG selectable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-260) An alternative approach to RNG
management
Posted by "Benson Margulies (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benson Margulies updated MAHOUT-260:
------------------------------------
Attachment: MAHOUT-260.diff
> An alternative approach to RNG management
> -----------------------------------------
>
> Key: MAHOUT-260
> URL: https://issues.apache.org/jira/browse/MAHOUT-260
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.3
> Reporter: Benson Margulies
> Attachments: MAHOUT-260.diff
>
>
> Here's code to (a) use the JDK for random numbers when testing, and (b) make the RNG selectable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAHOUT-260) An alternative approach to RNG
management
Posted by "Benson Margulies (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benson Margulies resolved MAHOUT-260.
-------------------------------------
Resolution: Not A Problem
Resolving as a dumb idea.
> An alternative approach to RNG management
> -----------------------------------------
>
> Key: MAHOUT-260
> URL: https://issues.apache.org/jira/browse/MAHOUT-260
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.3
> Reporter: Benson Margulies
> Fix For: 0.3
>
> Attachments: MAHOUT-260.diff, MAHOUT-260_wrapper.patch
>
>
> Here's code to (a) use the JDK for random numbers when testing, and (b) make the RNG selectable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-260) An alternative approach to RNG
management
Posted by "Ted Dunning (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801676#action_12801676 ]
Ted Dunning commented on MAHOUT-260:
------------------------------------
Pretty fancy.
I would prefer direct injection of the generator, but that would make the changes rather wide ranging and would require thought.
> An alternative approach to RNG management
> -----------------------------------------
>
> Key: MAHOUT-260
> URL: https://issues.apache.org/jira/browse/MAHOUT-260
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.3
> Reporter: Benson Margulies
> Attachments: MAHOUT-260.diff
>
>
> Here's code to (a) use the JDK for random numbers when testing, and (b) make the RNG selectable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-260) An alternative approach to RNG
management
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801750#action_12801750 ]
Sean Owen commented on MAHOUT-260:
----------------------------------
My take is that we have injection already, via RandomUtils. If the question is, can we do it more nicely? then I'd say the best hope is Guice, and the minimum effort required to use Guice is to rewrite every class that uses a RNG to take it as a constructor param, and add Guice annotations, and a config file, and Guice dependency. It doesn't change the functionality, and seems like work to go from something slightly complex, to complex.
But my big point here is none of that is actually addressing the real issue here, which is that even with injected RNGs, we have a test repeatability problem. My patch is attacking that issue, which exists with or without injection.
And actually even this change doesn't guarantee determinism, since we can't control randomness in libraries we use, though there may be none at the moment.
> An alternative approach to RNG management
> -----------------------------------------
>
> Key: MAHOUT-260
> URL: https://issues.apache.org/jira/browse/MAHOUT-260
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.3
> Reporter: Benson Margulies
> Attachments: MAHOUT-260.diff, MAHOUT-260_wrapper.patch
>
>
> Here's code to (a) use the JDK for random numbers when testing, and (b) make the RNG selectable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-260) An alternative approach to RNG
management
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803460#action_12803460 ]
Sean Owen commented on MAHOUT-260:
----------------------------------
Any thoughts on my patch? It makes a little mess of the Random handling, but it does in fact address the repeatability issue more correctly. But, it's a problem we have only in theory rather than practice right now.
Otherwise, can we close this?
> An alternative approach to RNG management
> -----------------------------------------
>
> Key: MAHOUT-260
> URL: https://issues.apache.org/jira/browse/MAHOUT-260
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.3
> Reporter: Benson Margulies
> Fix For: 0.3
>
> Attachments: MAHOUT-260.diff, MAHOUT-260_wrapper.patch
>
>
> Here's code to (a) use the JDK for random numbers when testing, and (b) make the RNG selectable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAHOUT-260) An alternative approach to RNG
management
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved MAHOUT-260.
------------------------------
Resolution: Fixed
This was committed a while ago no?
> An alternative approach to RNG management
> -----------------------------------------
>
> Key: MAHOUT-260
> URL: https://issues.apache.org/jira/browse/MAHOUT-260
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.3
> Reporter: Benson Margulies
> Assignee: Sean Owen
> Priority: Minor
> Attachments: MAHOUT-260.diff, MAHOUT-260_wrapper.patch
>
>
> Here's code to (a) use the JDK for random numbers when testing, and (b) make the RNG selectable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (MAHOUT-260) An alternative approach to RNG
management
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen reopened MAHOUT-260:
------------------------------
> An alternative approach to RNG management
> -----------------------------------------
>
> Key: MAHOUT-260
> URL: https://issues.apache.org/jira/browse/MAHOUT-260
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.3
> Reporter: Benson Margulies
> Assignee: Sean Owen
> Priority: Minor
> Attachments: MAHOUT-260.diff, MAHOUT-260_wrapper.patch
>
>
> Here's code to (a) use the JDK for random numbers when testing, and (b) make the RNG selectable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAHOUT-260) An alternative approach to RNG
management
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated MAHOUT-260:
-----------------------------
Fix Version/s: 0.3
> An alternative approach to RNG management
> -----------------------------------------
>
> Key: MAHOUT-260
> URL: https://issues.apache.org/jira/browse/MAHOUT-260
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.3
> Reporter: Benson Margulies
> Fix For: 0.3
>
> Attachments: MAHOUT-260.diff, MAHOUT-260_wrapper.patch
>
>
> Here's code to (a) use the JDK for random numbers when testing, and (b) make the RNG selectable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-260) An alternative approach to RNG
management
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12835056#action_12835056 ]
Sean Owen commented on MAHOUT-260:
----------------------------------
Great question -- it is only because there is no "WeakHashSet" in the JDK. I need to store references to these wrapper objects in a way that won't prevent them from begin GC'ed, and WeakHashMap is the best the JDK offers short of rolling our own.
> An alternative approach to RNG management
> -----------------------------------------
>
> Key: MAHOUT-260
> URL: https://issues.apache.org/jira/browse/MAHOUT-260
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.3
> Reporter: Benson Margulies
> Assignee: Sean Owen
> Priority: Minor
> Attachments: MAHOUT-260.diff, MAHOUT-260_wrapper.patch
>
>
> Here's code to (a) use the JDK for random numbers when testing, and (b) make the RNG selectable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAHOUT-260) An alternative approach to RNG
management
Posted by "Benson Margulies (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801748#action_12801748 ]
Benson Margulies commented on MAHOUT-260:
-----------------------------------------
Well,
I thought I saw email go by to the effect that this should all be configurable. I'm not attached to this, and it would be easy enough to put the Mersenne class back for the testSeed case.
Mostly I agree with Ted about getting rid of statics in favor of injection. Perhaps we should focus on that.
> An alternative approach to RNG management
> -----------------------------------------
>
> Key: MAHOUT-260
> URL: https://issues.apache.org/jira/browse/MAHOUT-260
> Project: Mahout
> Issue Type: Improvement
> Affects Versions: 0.3
> Reporter: Benson Margulies
> Attachments: MAHOUT-260.diff, MAHOUT-260_wrapper.patch
>
>
> Here's code to (a) use the JDK for random numbers when testing, and (b) make the RNG selectable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.