You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Jonathan Traupman (JIRA)" <ji...@apache.org> on 2011/03/29 20:38:05 UTC
[jira] [Created] (MAHOUT-641) DistributedRowMatrix hadoop jobs
ignore Configuration set via setConf()
DistributedRowMatrix hadoop jobs ignore Configuration set via setConf()
-----------------------------------------------------------------------
Key: MAHOUT-641
URL: https://issues.apache.org/jira/browse/MAHOUT-641
Project: Mahout
Issue Type: Bug
Components: Math
Affects Versions: 0.4, 0.5
Environment: Mahout 0.4 and 0.5-SNAPSHOT run with Hadoop 0.20.2 on Mac OS 10.6 and Linux x86_64 2.6.18
Reporter: Jonathan Traupman
I am using the Distributed Lanczos solver which uses the DistributedRowMatrix class for it's internal calculation. In our environment, I need to set some Configuration properties (specifically hadoop.job.ugi & hadoop.queue.name), which are when set via the setConf() method on DistributedLanczosSolver. These are correctly passed to DistributedRowMatrix via its setConf() method, but are not passed into the Hadoop JobConfs created by the various static routines in MatrixMultiplicationJob, TimesSquaredJob, and TransposeJob.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Updated] (MAHOUT-641) DistributedRowMatrix hadoop jobs
ignore Configuration set via setConf()
Posted by Ted Dunning <te...@gmail.com>.
Nice work Jonathan!
On Tue, Mar 29, 2011 at 11:42 AM, Jonathan Traupman (JIRA)
<ji...@apache.org>wrote:
> All unit tests pass. Added 3 new unit test cases to verify this bugfix.)
>
Re: [jira] [Commented] (MAHOUT-641) DistributedRowMatrix hadoop jobs
ignore Configuration set via setConf()
Posted by Lance Norskog <go...@gmail.com>.
If you want this to work smoothly, ban: 'new Job(conf)'. It's a point
source of head-scratching errors.
On Thu, Mar 31, 2011 at 4:32 PM, Hudson (JIRA) <ji...@apache.org> wrote:
>
> [ https://issues.apache.org/jira/browse/MAHOUT-641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014281#comment-13014281 ]
>
> Hudson commented on MAHOUT-641:
> -------------------------------
>
> Integrated in Mahout-Quality #708 (See [https://hudson.apache.org/hudson/job/Mahout-Quality/708/])
>
>
>> DistributedRowMatrix hadoop jobs ignore Configuration set via setConf()
>> -----------------------------------------------------------------------
>>
>> Key: MAHOUT-641
>> URL: https://issues.apache.org/jira/browse/MAHOUT-641
>> Project: Mahout
>> Issue Type: Bug
>> Components: Math
>> Affects Versions: 0.4, 0.5
>> Environment: Mahout 0.4 and 0.5-SNAPSHOT run with Hadoop 0.20.2 on Mac OS 10.6 and Linux x86_64 2.6.18
>> Reporter: Jonathan Traupman
>> Labels: configuration, hadoop
>> Fix For: 0.5
>>
>> Attachments: MAHOUT-641.patch
>>
>> Original Estimate: 24h
>> Remaining Estimate: 24h
>>
>> I am using the Distributed Lanczos solver which uses the DistributedRowMatrix class for it's internal calculation. In our environment, I need to set some Configuration properties (specifically hadoop.job.ugi & hadoop.queue.name), which are when set via the setConf() method on DistributedLanczosSolver. These are correctly passed to DistributedRowMatrix via its setConf() method, but are not passed into the Hadoop JobConfs created by the various static routines in MatrixMultiplicationJob, TimesSquaredJob, and TransposeJob.
>
> --
> This message is automatically generated by JIRA.
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
--
Lance Norskog
goksron@gmail.com
[jira] [Updated] (MAHOUT-641) DistributedRowMatrix hadoop jobs
ignore Configuration set via setConf()
Posted by "Jonathan Traupman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Traupman updated MAHOUT-641:
-------------------------------------
Attachment: MAHOUT-641.patch
Patch for MAHOUT-641. Diffed from revision 1086678.
> DistributedRowMatrix hadoop jobs ignore Configuration set via setConf()
> -----------------------------------------------------------------------
>
> Key: MAHOUT-641
> URL: https://issues.apache.org/jira/browse/MAHOUT-641
> Project: Mahout
> Issue Type: Bug
> Components: Math
> Affects Versions: 0.4, 0.5
> Environment: Mahout 0.4 and 0.5-SNAPSHOT run with Hadoop 0.20.2 on Mac OS 10.6 and Linux x86_64 2.6.18
> Reporter: Jonathan Traupman
> Labels: configuration, hadoop
> Fix For: 0.5
>
> Attachments: MAHOUT-641.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> I am using the Distributed Lanczos solver which uses the DistributedRowMatrix class for it's internal calculation. In our environment, I need to set some Configuration properties (specifically hadoop.job.ugi & hadoop.queue.name), which are when set via the setConf() method on DistributedLanczosSolver. These are correctly passed to DistributedRowMatrix via its setConf() method, but are not passed into the Hadoop JobConfs created by the various static routines in MatrixMultiplicationJob, TimesSquaredJob, and TransposeJob.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-641) DistributedRowMatrix hadoop jobs
ignore Configuration set via setConf()
Posted by "Jonathan Traupman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Traupman updated MAHOUT-641:
-------------------------------------
Status: Patch Available (was: Open)
Sorry for the duplicate message -- still figuring out the patch submission process.
Anyway, the patch is uploaded, all existing tests pass and I added 3 new cases for this specific bug. The patch fixes the problems I was having in our environment.
> DistributedRowMatrix hadoop jobs ignore Configuration set via setConf()
> -----------------------------------------------------------------------
>
> Key: MAHOUT-641
> URL: https://issues.apache.org/jira/browse/MAHOUT-641
> Project: Mahout
> Issue Type: Bug
> Components: Math
> Affects Versions: 0.4, 0.5
> Environment: Mahout 0.4 and 0.5-SNAPSHOT run with Hadoop 0.20.2 on Mac OS 10.6 and Linux x86_64 2.6.18
> Reporter: Jonathan Traupman
> Labels: configuration, hadoop
> Fix For: 0.5
>
> Attachments: MAHOUT-641.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> I am using the Distributed Lanczos solver which uses the DistributedRowMatrix class for it's internal calculation. In our environment, I need to set some Configuration properties (specifically hadoop.job.ugi & hadoop.queue.name), which are when set via the setConf() method on DistributedLanczosSolver. These are correctly passed to DistributedRowMatrix via its setConf() method, but are not passed into the Hadoop JobConfs created by the various static routines in MatrixMultiplicationJob, TimesSquaredJob, and TransposeJob.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-641) DistributedRowMatrix hadoop jobs
ignore Configuration set via setConf()
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013912#comment-13013912 ]
Sean Owen commented on MAHOUT-641:
----------------------------------
I'll commit it. More broadly I think there is some inconsistency in the code about who passes a Configuration when, and a lot of code ignores or makes its own Configuration. Configuration is the context of Hadoop operations, and probably needs to be passed around most everywhere in Mahout (rather than be created anew). The code probably "gets away" with not doing it in several cases, but not this one.
I cleaned up a small version of this in MAHOUT-633, having to do with obtaining a FileSystem to delete files properly. It's a broader issue and not one to solve right now perhaps.
> DistributedRowMatrix hadoop jobs ignore Configuration set via setConf()
> -----------------------------------------------------------------------
>
> Key: MAHOUT-641
> URL: https://issues.apache.org/jira/browse/MAHOUT-641
> Project: Mahout
> Issue Type: Bug
> Components: Math
> Affects Versions: 0.4, 0.5
> Environment: Mahout 0.4 and 0.5-SNAPSHOT run with Hadoop 0.20.2 on Mac OS 10.6 and Linux x86_64 2.6.18
> Reporter: Jonathan Traupman
> Labels: configuration, hadoop
> Fix For: 0.5
>
> Attachments: MAHOUT-641.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> I am using the Distributed Lanczos solver which uses the DistributedRowMatrix class for it's internal calculation. In our environment, I need to set some Configuration properties (specifically hadoop.job.ugi & hadoop.queue.name), which are when set via the setConf() method on DistributedLanczosSolver. These are correctly passed to DistributedRowMatrix via its setConf() method, but are not passed into the Hadoop JobConfs created by the various static routines in MatrixMultiplicationJob, TimesSquaredJob, and TransposeJob.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-641) DistributedRowMatrix hadoop jobs
ignore Configuration set via setConf()
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13014281#comment-13014281 ]
Hudson commented on MAHOUT-641:
-------------------------------
Integrated in Mahout-Quality #708 (See [https://hudson.apache.org/hudson/job/Mahout-Quality/708/])
> DistributedRowMatrix hadoop jobs ignore Configuration set via setConf()
> -----------------------------------------------------------------------
>
> Key: MAHOUT-641
> URL: https://issues.apache.org/jira/browse/MAHOUT-641
> Project: Mahout
> Issue Type: Bug
> Components: Math
> Affects Versions: 0.4, 0.5
> Environment: Mahout 0.4 and 0.5-SNAPSHOT run with Hadoop 0.20.2 on Mac OS 10.6 and Linux x86_64 2.6.18
> Reporter: Jonathan Traupman
> Labels: configuration, hadoop
> Fix For: 0.5
>
> Attachments: MAHOUT-641.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> I am using the Distributed Lanczos solver which uses the DistributedRowMatrix class for it's internal calculation. In our environment, I need to set some Configuration properties (specifically hadoop.job.ugi & hadoop.queue.name), which are when set via the setConf() method on DistributedLanczosSolver. These are correctly passed to DistributedRowMatrix via its setConf() method, but are not passed into the Hadoop JobConfs created by the various static routines in MatrixMultiplicationJob, TimesSquaredJob, and TransposeJob.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-641) DistributedRowMatrix hadoop jobs
ignore Configuration set via setConf()
Posted by "Jonathan Traupman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Traupman updated MAHOUT-641:
-------------------------------------
Status: Open (was: Patch Available)
> DistributedRowMatrix hadoop jobs ignore Configuration set via setConf()
> -----------------------------------------------------------------------
>
> Key: MAHOUT-641
> URL: https://issues.apache.org/jira/browse/MAHOUT-641
> Project: Mahout
> Issue Type: Bug
> Components: Math
> Affects Versions: 0.4, 0.5
> Environment: Mahout 0.4 and 0.5-SNAPSHOT run with Hadoop 0.20.2 on Mac OS 10.6 and Linux x86_64 2.6.18
> Reporter: Jonathan Traupman
> Labels: configuration, hadoop
> Fix For: 0.5
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> I am using the Distributed Lanczos solver which uses the DistributedRowMatrix class for it's internal calculation. In our environment, I need to set some Configuration properties (specifically hadoop.job.ugi & hadoop.queue.name), which are when set via the setConf() method on DistributedLanczosSolver. These are correctly passed to DistributedRowMatrix via its setConf() method, but are not passed into the Hadoop JobConfs created by the various static routines in MatrixMultiplicationJob, TimesSquaredJob, and TransposeJob.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-641) DistributedRowMatrix hadoop jobs
ignore Configuration set via setConf()
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated MAHOUT-641:
-----------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> DistributedRowMatrix hadoop jobs ignore Configuration set via setConf()
> -----------------------------------------------------------------------
>
> Key: MAHOUT-641
> URL: https://issues.apache.org/jira/browse/MAHOUT-641
> Project: Mahout
> Issue Type: Bug
> Components: Math
> Affects Versions: 0.4, 0.5
> Environment: Mahout 0.4 and 0.5-SNAPSHOT run with Hadoop 0.20.2 on Mac OS 10.6 and Linux x86_64 2.6.18
> Reporter: Jonathan Traupman
> Labels: configuration, hadoop
> Fix For: 0.5
>
> Attachments: MAHOUT-641.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> I am using the Distributed Lanczos solver which uses the DistributedRowMatrix class for it's internal calculation. In our environment, I need to set some Configuration properties (specifically hadoop.job.ugi & hadoop.queue.name), which are when set via the setConf() method on DistributedLanczosSolver. These are correctly passed to DistributedRowMatrix via its setConf() method, but are not passed into the Hadoop JobConfs created by the various static routines in MatrixMultiplicationJob, TimesSquaredJob, and TransposeJob.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-641) DistributedRowMatrix hadoop jobs
ignore Configuration set via setConf()
Posted by "Ted Dunning (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012611#comment-13012611 ]
Ted Dunning commented on MAHOUT-641:
------------------------------------
Nice work Jonathan!
On Tue, Mar 29, 2011 at 11:42 AM, Jonathan Traupman (JIRA)
> DistributedRowMatrix hadoop jobs ignore Configuration set via setConf()
> -----------------------------------------------------------------------
>
> Key: MAHOUT-641
> URL: https://issues.apache.org/jira/browse/MAHOUT-641
> Project: Mahout
> Issue Type: Bug
> Components: Math
> Affects Versions: 0.4, 0.5
> Environment: Mahout 0.4 and 0.5-SNAPSHOT run with Hadoop 0.20.2 on Mac OS 10.6 and Linux x86_64 2.6.18
> Reporter: Jonathan Traupman
> Labels: configuration, hadoop
> Fix For: 0.5
>
> Attachments: MAHOUT-641.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> I am using the Distributed Lanczos solver which uses the DistributedRowMatrix class for it's internal calculation. In our environment, I need to set some Configuration properties (specifically hadoop.job.ugi & hadoop.queue.name), which are when set via the setConf() method on DistributedLanczosSolver. These are correctly passed to DistributedRowMatrix via its setConf() method, but are not passed into the Hadoop JobConfs created by the various static routines in MatrixMultiplicationJob, TimesSquaredJob, and TransposeJob.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-641) DistributedRowMatrix hadoop jobs
ignore Configuration set via setConf()
Posted by "Jonathan Traupman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Traupman updated MAHOUT-641:
-------------------------------------
Fix Version/s: 0.5
Status: Patch Available (was: Open)
Patch fixes configuration problem in our environment. I changed the existing static JobConf creation routines to take an initial Configuration that will be passed in by DistributedRowMatrix if specified. I also included backward-compatible static routines that maintain the existing signature and behavior.
All unit tests pass. Added 3 new unit test cases to verify this bugfix.
> DistributedRowMatrix hadoop jobs ignore Configuration set via setConf()
> -----------------------------------------------------------------------
>
> Key: MAHOUT-641
> URL: https://issues.apache.org/jira/browse/MAHOUT-641
> Project: Mahout
> Issue Type: Bug
> Components: Math
> Affects Versions: 0.4, 0.5
> Environment: Mahout 0.4 and 0.5-SNAPSHOT run with Hadoop 0.20.2 on Mac OS 10.6 and Linux x86_64 2.6.18
> Reporter: Jonathan Traupman
> Labels: configuration, hadoop
> Fix For: 0.5
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> I am using the Distributed Lanczos solver which uses the DistributedRowMatrix class for it's internal calculation. In our environment, I need to set some Configuration properties (specifically hadoop.job.ugi & hadoop.queue.name), which are when set via the setConf() method on DistributedLanczosSolver. These are correctly passed to DistributedRowMatrix via its setConf() method, but are not passed into the Hadoop JobConfs created by the various static routines in MatrixMultiplicationJob, TimesSquaredJob, and TransposeJob.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-641) DistributedRowMatrix hadoop jobs
ignore Configuration set via setConf()
Posted by "Jonathan Traupman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Traupman updated MAHOUT-641:
-------------------------------------
Comment: was deleted
(was: Patch fixes configuration problem in our environment. I changed the existing static JobConf creation routines to take an initial Configuration that will be passed in by DistributedRowMatrix if specified. I also included backward-compatible static routines that maintain the existing signature and behavior.
All unit tests pass. Added 3 new unit test cases to verify this bugfix.)
> DistributedRowMatrix hadoop jobs ignore Configuration set via setConf()
> -----------------------------------------------------------------------
>
> Key: MAHOUT-641
> URL: https://issues.apache.org/jira/browse/MAHOUT-641
> Project: Mahout
> Issue Type: Bug
> Components: Math
> Affects Versions: 0.4, 0.5
> Environment: Mahout 0.4 and 0.5-SNAPSHOT run with Hadoop 0.20.2 on Mac OS 10.6 and Linux x86_64 2.6.18
> Reporter: Jonathan Traupman
> Labels: configuration, hadoop
> Fix For: 0.5
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> I am using the Distributed Lanczos solver which uses the DistributedRowMatrix class for it's internal calculation. In our environment, I need to set some Configuration properties (specifically hadoop.job.ugi & hadoop.queue.name), which are when set via the setConf() method on DistributedLanczosSolver. These are correctly passed to DistributedRowMatrix via its setConf() method, but are not passed into the Hadoop JobConfs created by the various static routines in MatrixMultiplicationJob, TimesSquaredJob, and TransposeJob.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira