You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Grant Ingersoll (Created) (JIRA)" <ji...@apache.org> on 2011/12/06 16:19:39 UTC
[jira] [Created] (MAHOUT-916) Make Mahout's tests run in parallel
Make Mahout's tests run in parallel
-----------------------------------
Key: MAHOUT-916
URL: https://issues.apache.org/jira/browse/MAHOUT-916
Project: Mahout
Issue Type: Improvement
Components: build
Reporter: Grant Ingersoll
Priority: Minor
Maven now supports parallel execution of tests. We should hook this in to Mahout.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-916) Make Mahout's tests run in parallel
Posted by "Grant Ingersoll (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13166253#comment-13166253 ]
Grant Ingersoll commented on MAHOUT-916:
----------------------------------------
On that note, I suppose it's due to the fact that thread count is per forked JVM, so you really don't see much benefit. I wonder, if we could get back to forking only once, if we would then see some real benefit?
> Make Mahout's tests run in parallel
> -----------------------------------
>
> Key: MAHOUT-916
> URL: https://issues.apache.org/jira/browse/MAHOUT-916
> Project: Mahout
> Issue Type: Improvement
> Components: build
> Reporter: Grant Ingersoll
> Priority: Minor
> Labels: MAHOUT_INTRO_CONTRIBUTE
> Attachments: MAHOUT-916.patch
>
>
> Maven now supports parallel execution of tests. We should hook this in to Mahout.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-916) Make Mahout's tests run in parallel
Posted by "Grant Ingersoll (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Grant Ingersoll updated MAHOUT-916:
-----------------------------------
Attachment: MAHOUT-916.patch
worth trying out.
> Make Mahout's tests run in parallel
> -----------------------------------
>
> Key: MAHOUT-916
> URL: https://issues.apache.org/jira/browse/MAHOUT-916
> Project: Mahout
> Issue Type: Improvement
> Components: build
> Reporter: Grant Ingersoll
> Priority: Minor
> Labels: MAHOUT_INTRO_CONTRIBUTE
> Attachments: MAHOUT-916.patch
>
>
> Maven now supports parallel execution of tests. We should hook this in to Mahout.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-916) Make Mahout's tests run in parallel
Posted by "Grant Ingersoll (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13166229#comment-13166229 ]
Grant Ingersoll commented on MAHOUT-916:
----------------------------------------
Yeah, not sure it's much savings. One thing I wonder about is the use of "balanced". I think, in theory, the benefit of this comes from running the tests multiple times, as Maven will collect stats about what runs take longer.
> Make Mahout's tests run in parallel
> -----------------------------------
>
> Key: MAHOUT-916
> URL: https://issues.apache.org/jira/browse/MAHOUT-916
> Project: Mahout
> Issue Type: Improvement
> Components: build
> Reporter: Grant Ingersoll
> Priority: Minor
> Labels: MAHOUT_INTRO_CONTRIBUTE
> Attachments: MAHOUT-916.patch
>
>
> Maven now supports parallel execution of tests. We should hook this in to Mahout.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-916) Make Mahout's tests run in parallel
Posted by "Grant Ingersoll (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13166247#comment-13166247 ]
Grant Ingersoll commented on MAHOUT-916:
----------------------------------------
The other weird thing is even when I run this, w/ thread count of 4 per core, it barely touches the CPU. I guess I would expect that it would be pushing it more.
> Make Mahout's tests run in parallel
> -----------------------------------
>
> Key: MAHOUT-916
> URL: https://issues.apache.org/jira/browse/MAHOUT-916
> Project: Mahout
> Issue Type: Improvement
> Components: build
> Reporter: Grant Ingersoll
> Priority: Minor
> Labels: MAHOUT_INTRO_CONTRIBUTE
> Attachments: MAHOUT-916.patch
>
>
> Maven now supports parallel execution of tests. We should hook this in to Mahout.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-916) Make Mahout's tests run in parallel
Posted by "Sean Owen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13166043#comment-13166043 ]
Sean Owen commented on MAHOUT-916:
----------------------------------
The patch works for me. However it takes just about the same time for me -- one hour.
I see this message on every test:
2011-12-09 10:02:26.194 java[10996:ad03] Unable to load realm info from SCDynamicStore
but I think it's unrelated:
https://issues.apache.org/jira/browse/HADOOP-7489
I think the config is quite right though. Is anyone seeing a speedup -- is it perhaps an artifact of my local environment?
> Make Mahout's tests run in parallel
> -----------------------------------
>
> Key: MAHOUT-916
> URL: https://issues.apache.org/jira/browse/MAHOUT-916
> Project: Mahout
> Issue Type: Improvement
> Components: build
> Reporter: Grant Ingersoll
> Priority: Minor
> Labels: MAHOUT_INTRO_CONTRIBUTE
> Attachments: MAHOUT-916.patch
>
>
> Maven now supports parallel execution of tests. We should hook this in to Mahout.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-916) Make Mahout's tests run in parallel
Posted by "Sean Owen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13166325#comment-13166325 ]
Sean Owen commented on MAHOUT-916:
----------------------------------
That is what I see. I am not even sure this is forking 4 JVMs. The "perCoreThreadCount" setting doesn't help either. I think there is just not quite a setting for what we want to do in this ticket.
> Make Mahout's tests run in parallel
> -----------------------------------
>
> Key: MAHOUT-916
> URL: https://issues.apache.org/jira/browse/MAHOUT-916
> Project: Mahout
> Issue Type: Improvement
> Components: build
> Reporter: Grant Ingersoll
> Priority: Minor
> Labels: MAHOUT_INTRO_CONTRIBUTE
> Attachments: MAHOUT-916.patch
>
>
> Maven now supports parallel execution of tests. We should hook this in to Mahout.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAHOUT-916) Make Mahout's tests run in parallel
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved MAHOUT-916.
------------------------------
Resolution: Won't Fix
Assignee: Sean Owen
More like "CantFix" -- we both never found any way to run in parallel that worked correcty.
> Make Mahout's tests run in parallel
> -----------------------------------
>
> Key: MAHOUT-916
> URL: https://issues.apache.org/jira/browse/MAHOUT-916
> Project: Mahout
> Issue Type: Improvement
> Components: build
> Reporter: Grant Ingersoll
> Assignee: Sean Owen
> Priority: Minor
> Labels: MAHOUT_INTRO_CONTRIBUTE
> Attachments: MAHOUT-916.patch
>
>
> Maven now supports parallel execution of tests. We should hook this in to Mahout.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira