You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "tom pierce (Created) (JIRA)" <ji...@apache.org> on 2012/01/15 23:41:41 UTC
[jira] [Created] (MAHOUT-946) Map-reduce job status often left
unchecked
Map-reduce job status often left unchecked
------------------------------------------
Key: MAHOUT-946
URL: https://issues.apache.org/jira/browse/MAHOUT-946
Project: Mahout
Issue Type: Bug
Reporter: tom pierce
I've run into a few places in Mahout where mapreduce jobs can fail and their status won't be checked, so processing will continue on. This sometimes obscures the root problem. I've tracked down a bunch of places where this problem exists, and tried to fix the code to do something reasonable (return -1 status code, throw exception, etc.) when jobs fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-946) Map-reduce job status often left
unchecked
Posted by "Lance Norskog (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187369#comment-13187369 ]
Lance Norskog commented on MAHOUT-946:
--------------------------------------
Yup, you're right. Shell script should be a separate project.
One thing that jumps out at me is refactoring. The AbstractJob class is a toolkit for common tasks. Right not it has a lot of kit for maybe the first 1/3 of the code to set up and run a job from the command line. Could it also have a "run job and fail nicely" method?
> Map-reduce job status often left unchecked
> ------------------------------------------
>
> Key: MAHOUT-946
> URL: https://issues.apache.org/jira/browse/MAHOUT-946
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.6
> Reporter: tom pierce
> Attachments: MAHOUT-946.patch
>
>
> I've run into a few places in Mahout where mapreduce jobs can fail and their status won't be checked, so processing will continue on. This sometimes obscures the root problem. I've tracked down a bunch of places where this problem exists, and tried to fix the code to do something reasonable (return -1 status code, throw exception, etc.) when jobs fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-946) Map-reduce job status often left
unchecked
Posted by "tom pierce (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203694#comment-13203694 ]
tom pierce commented on MAHOUT-946:
-----------------------------------
I agree that the cleanup stuff seems a bit awkward as is.
I'd like to see the checks get in, and I think halting execution on job failure is something folks can agree is an improvement. It might be harder to settle on consensus for job cleanup strategies.
Would it make sense at this point to integrate the earlier patch, without the cleanups, and split off utility methods for cleanup to another ticket? If so I'll volunteer to open a new issue and submit an initial patch to address the cleanup issue.
> Map-reduce job status often left unchecked
> ------------------------------------------
>
> Key: MAHOUT-946
> URL: https://issues.apache.org/jira/browse/MAHOUT-946
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.6
> Reporter: tom pierce
> Attachments: MAHOUT-946.patch, MAHOUT-946.patch
>
>
> I've run into a few places in Mahout where mapreduce jobs can fail and their status won't be checked, so processing will continue on. This sometimes obscures the root problem. I've tracked down a bunch of places where this problem exists, and tried to fix the code to do something reasonable (return -1 status code, throw exception, etc.) when jobs fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-946) Map-reduce job status often left
unchecked
Posted by "Sean Owen (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated MAHOUT-946:
-----------------------------
Resolution: Fixed
Fix Version/s: 0.7
Assignee: Sean Owen
Status: Resolved (was: Patch Available)
Agree, that much is done and committed now.
> Map-reduce job status often left unchecked
> ------------------------------------------
>
> Key: MAHOUT-946
> URL: https://issues.apache.org/jira/browse/MAHOUT-946
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.6
> Reporter: tom pierce
> Assignee: Sean Owen
> Fix For: 0.7
>
> Attachments: MAHOUT-946.patch, MAHOUT-946.patch
>
>
> I've run into a few places in Mahout where mapreduce jobs can fail and their status won't be checked, so processing will continue on. This sometimes obscures the root problem. I've tracked down a bunch of places where this problem exists, and tried to fix the code to do something reasonable (return -1 status code, throw exception, etc.) when jobs fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-946) Map-reduce job status often left
unchecked
Posted by "tom pierce (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187264#comment-13187264 ]
tom pierce commented on MAHOUT-946:
-----------------------------------
Making sure the examples halt at appropriate places looks like it could be pretty involved. For example, the first program where I added exit-code checks (asf-email-examples.sh) would not fail early, even when given a non-existant input directory. Digging in, I found the java code (SequenceFilesFromMailArchives and MailProcessor) is happy to skip "missing" files during processing, even if that leads to 0 input records overall.
I'm not sure I have the time/bandwidth to audit all of these. Can we split that off into another JIRA (or maybe several for different examples)?
> Map-reduce job status often left unchecked
> ------------------------------------------
>
> Key: MAHOUT-946
> URL: https://issues.apache.org/jira/browse/MAHOUT-946
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.6
> Reporter: tom pierce
> Attachments: MAHOUT-946.patch
>
>
> I've run into a few places in Mahout where mapreduce jobs can fail and their status won't be checked, so processing will continue on. This sometimes obscures the root problem. I've tracked down a bunch of places where this problem exists, and tried to fix the code to do something reasonable (return -1 status code, throw exception, etc.) when jobs fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-946) Map-reduce job status often left
unchecked
Posted by "tom pierce (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
tom pierce updated MAHOUT-946:
------------------------------
Affects Version/s: 0.6
Status: Patch Available (was: Open)
> Map-reduce job status often left unchecked
> ------------------------------------------
>
> Key: MAHOUT-946
> URL: https://issues.apache.org/jira/browse/MAHOUT-946
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.6
> Reporter: tom pierce
> Attachments: MAHOUT-946.patch
>
>
> I've run into a few places in Mahout where mapreduce jobs can fail and their status won't be checked, so processing will continue on. This sometimes obscures the root problem. I've tracked down a bunch of places where this problem exists, and tried to fix the code to do something reasonable (return -1 status code, throw exception, etc.) when jobs fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-946) Map-reduce job status often left
unchecked
Posted by "Lance Norskog (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195416#comment-13195416 ]
Lance Norskog commented on MAHOUT-946:
--------------------------------------
Sweet!
> Map-reduce job status often left unchecked
> ------------------------------------------
>
> Key: MAHOUT-946
> URL: https://issues.apache.org/jira/browse/MAHOUT-946
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.6
> Reporter: tom pierce
> Attachments: MAHOUT-946.patch, MAHOUT-946.patch
>
>
> I've run into a few places in Mahout where mapreduce jobs can fail and their status won't be checked, so processing will continue on. This sometimes obscures the root problem. I've tracked down a bunch of places where this problem exists, and tried to fix the code to do something reasonable (return -1 status code, throw exception, etc.) when jobs fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (MAHOUT-946) Map-reduce job status
often left unchecked
Posted by "Dmitriy Lyubimov (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202917#comment-13202917 ]
Dmitriy Lyubimov edited comment on MAHOUT-946 at 2/7/12 11:08 PM:
------------------------------------------------------------------
bq. Some of these classes are also not AbstractJobs (though in some cases they should probably be converted - not sure it would make sense in all cases), some return some non-status value from the method that fires of the MR job, etc. Maybe a static util method would make more sense?
Static util method would probably make more sense.
My current view of AbstractJob is that it is mixing two disparate concerns in one implementation: a Tool (pipeline bootstrap) in a sense of handing off the default Configuration, to which end it has a lot of helper methods, and a distinct MR job. Due to this, it is hard (or at least inconsistent) to use it as a base for a separate and highly specific step in multistep pipelines.
Static method would not require AbstractJob as a base, so static drop-in is probably a better approach at this point.
was (Author: dlyubimov):
bq. Some of these classes are also not AbstractJobs (though in some cases they should probably be converted - not sure it would make sense in all cases), some return some non-status value from the method that fires of the MR job, etc. Maybe a static util method would make more sense?
Static util method would probably make more sense.
My current view of AbstractJob is that it mixing two disparate concerns in one implementation: a Tool (pipeline bootstrap) in a sense of handing off the default Configuration, to which end it has a lot of helper methods, and a distinct MR job. Due to this, it is hard (or at least inconsistent) to use it as a base for a separate and highly specific step in a multistep pipelines.
Static method would not require AbstractJob as a base, so static drop-in is probably a better approach at this point.
> Map-reduce job status often left unchecked
> ------------------------------------------
>
> Key: MAHOUT-946
> URL: https://issues.apache.org/jira/browse/MAHOUT-946
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.6
> Reporter: tom pierce
> Attachments: MAHOUT-946.patch, MAHOUT-946.patch
>
>
> I've run into a few places in Mahout where mapreduce jobs can fail and their status won't be checked, so processing will continue on. This sometimes obscures the root problem. I've tracked down a bunch of places where this problem exists, and tried to fix the code to do something reasonable (return -1 status code, throw exception, etc.) when jobs fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-946) Map-reduce job status often left
unchecked
Posted by "Sean Owen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202848#comment-13202848 ]
Sean Owen commented on MAHOUT-946:
----------------------------------
I like it all, except I'm not sure about the cleanup methods. I think as you say the job can manage that if it needs to clean up; at least, that's how it works now.
> Map-reduce job status often left unchecked
> ------------------------------------------
>
> Key: MAHOUT-946
> URL: https://issues.apache.org/jira/browse/MAHOUT-946
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.6
> Reporter: tom pierce
> Attachments: MAHOUT-946.patch, MAHOUT-946.patch
>
>
> I've run into a few places in Mahout where mapreduce jobs can fail and their status won't be checked, so processing will continue on. This sometimes obscures the root problem. I've tracked down a bunch of places where this problem exists, and tried to fix the code to do something reasonable (return -1 status code, throw exception, etc.) when jobs fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-946) Map-reduce job status often left
unchecked
Posted by "Lance Norskog (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203290#comment-13203290 ]
Lance Norskog commented on MAHOUT-946:
--------------------------------------
bq. My current view of AbstractJob is that it is mixing two disparate concerns in one implementation: a Tool (pipeline bootstrap) in a sense of handing off the default Configuration, to which end it has a lot of helper methods, and a distinct MR job
There are some single-threaded AbstractJob apps. It handles so much argument processing nonsense that it is useful for that alone.
> Map-reduce job status often left unchecked
> ------------------------------------------
>
> Key: MAHOUT-946
> URL: https://issues.apache.org/jira/browse/MAHOUT-946
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.6
> Reporter: tom pierce
> Attachments: MAHOUT-946.patch, MAHOUT-946.patch
>
>
> I've run into a few places in Mahout where mapreduce jobs can fail and their status won't be checked, so processing will continue on. This sometimes obscures the root problem. I've tracked down a bunch of places where this problem exists, and tried to fix the code to do something reasonable (return -1 status code, throw exception, etc.) when jobs fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-946) Map-reduce job status often left
unchecked
Posted by "Lance Norskog (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188152#comment-13188152 ]
Lance Norskog commented on MAHOUT-946:
--------------------------------------
A job shutdown method should remember to remove intermediate files, and if it fails remove the output. A job should be able to override the "run the job" method with extra cleanups.
> Map-reduce job status often left unchecked
> ------------------------------------------
>
> Key: MAHOUT-946
> URL: https://issues.apache.org/jira/browse/MAHOUT-946
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.6
> Reporter: tom pierce
> Attachments: MAHOUT-946.patch
>
>
> I've run into a few places in Mahout where mapreduce jobs can fail and their status won't be checked, so processing will continue on. This sometimes obscures the root problem. I've tracked down a bunch of places where this problem exists, and tried to fix the code to do something reasonable (return -1 status code, throw exception, etc.) when jobs fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (MAHOUT-946) Map-reduce job status
often left unchecked
Posted by "Lance Norskog (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187369#comment-13187369 ]
Lance Norskog edited comment on MAHOUT-946 at 1/17/12 1:47 AM:
---------------------------------------------------------------
Yup, you're right. Shell script should be a separate project.
One thing that jumps out at me is refactoring. The AbstractJob class is a toolkit for common tasks. It has a lot of kit for maybe the first 1/3 of the code to set up and run a job from the command line. Could it also have a "run job and fail nicely" method?
was (Author: lancenorskog):
Yup, you're right. Shell script should be a separate project.
One thing that jumps out at me is refactoring. The AbstractJob class is a toolkit for common tasks. Right not it has a lot of kit for maybe the first 1/3 of the code to set up and run a job from the command line. Could it also have a "run job and fail nicely" method?
> Map-reduce job status often left unchecked
> ------------------------------------------
>
> Key: MAHOUT-946
> URL: https://issues.apache.org/jira/browse/MAHOUT-946
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.6
> Reporter: tom pierce
> Attachments: MAHOUT-946.patch
>
>
> I've run into a few places in Mahout where mapreduce jobs can fail and their status won't be checked, so processing will continue on. This sometimes obscures the root problem. I've tracked down a bunch of places where this problem exists, and tried to fix the code to do something reasonable (return -1 status code, throw exception, etc.) when jobs fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-946) Map-reduce job status often left
unchecked
Posted by "tom pierce (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
tom pierce updated MAHOUT-946:
------------------------------
Attachment: MAHOUT-946.patch
I thought about this some more and realized in many cases you might want opposite behavior (delete the intermediate files unless something broke - in that case save them for analysis).
In the end, I added a way to accumulate a list of Paths and clean those paths at the end of processing, but left accumulation and triggering deletion as subclass responsibilities.
> Map-reduce job status often left unchecked
> ------------------------------------------
>
> Key: MAHOUT-946
> URL: https://issues.apache.org/jira/browse/MAHOUT-946
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.6
> Reporter: tom pierce
> Attachments: MAHOUT-946.patch, MAHOUT-946.patch
>
>
> I've run into a few places in Mahout where mapreduce jobs can fail and their status won't be checked, so processing will continue on. This sometimes obscures the root problem. I've tracked down a bunch of places where this problem exists, and tried to fix the code to do something reasonable (return -1 status code, throw exception, etc.) when jobs fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-946) Map-reduce job status often left
unchecked
Posted by "tom pierce (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
tom pierce updated MAHOUT-946:
------------------------------
Attachment: MAHOUT-946.patch
> Map-reduce job status often left unchecked
> ------------------------------------------
>
> Key: MAHOUT-946
> URL: https://issues.apache.org/jira/browse/MAHOUT-946
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.6
> Reporter: tom pierce
> Attachments: MAHOUT-946.patch
>
>
> I've run into a few places in Mahout where mapreduce jobs can fail and their status won't be checked, so processing will continue on. This sometimes obscures the root problem. I've tracked down a bunch of places where this problem exists, and tried to fix the code to do something reasonable (return -1 status code, throw exception, etc.) when jobs fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-946) Map-reduce job status often left
unchecked
Posted by "tom pierce (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187755#comment-13187755 ]
tom pierce commented on MAHOUT-946:
-----------------------------------
Interesting thought - right now it seems to me the simplest "drop in" convenience method would be a (better-named) runAndThrowExceptionOnFailure(). There could also be a version that returned a status code, but I think most locations that use status codes also run multiple jobs. So you still have to check the return value at many of the call-sites, just like with Job.waitForCompletion().
Some of these classes are also not AbstractJobs (though in some cases they should probably be converted - not sure it would make sense in all cases), some return some non-status value from the method that fires of the MR job, etc. Maybe a static util method would make more sense?
Taking a long view, I'd actually prefer to see most of these call sites re-worked to use return codes, rather than bake in exceptions. Probably not pragmatic, though. Sometimes done is the best color.
Absent objections in the next day or so, I'll do something along the lines of static Job.waitForCompletion wrappers in AbstractJob.
> Map-reduce job status often left unchecked
> ------------------------------------------
>
> Key: MAHOUT-946
> URL: https://issues.apache.org/jira/browse/MAHOUT-946
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.6
> Reporter: tom pierce
> Attachments: MAHOUT-946.patch
>
>
> I've run into a few places in Mahout where mapreduce jobs can fail and their status won't be checked, so processing will continue on. This sometimes obscures the root problem. I've tracked down a bunch of places where this problem exists, and tried to fix the code to do something reasonable (return -1 status code, throw exception, etc.) when jobs fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-946) Map-reduce job status often left
unchecked
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203986#comment-13203986 ]
Hudson commented on MAHOUT-946:
-------------------------------
Integrated in Mahout-Quality #1341 (See [https://builds.apache.org/job/Mahout-Quality/1341/])
MAHOUT-946 (missed one file)
MAHOUT-946 Don't ignore failed MR status
srowen : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241992
Files :
* /mahout/trunk/core/src/main/java/org/apache/mahout/classifier/naivebayes/test/TestNaiveBayesDriver.java
srowen : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241988
Files :
* /mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/hadoop/als/DatasetSplitter.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/hadoop/als/FactorizationEvaluator.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/hadoop/als/ParallelALSFactorizationJob.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/hadoop/als/RecommenderJob.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/hadoop/item/RecommenderJob.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/hadoop/preparation/PreparePreferenceMatrixJob.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/hadoop/pseudo/RecommenderJob.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/hadoop/similarity/item/ItemSimilarityJob.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/hadoop/slopeone/SlopeOneAverageDiffsJob.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/classifier/df/mapreduce/Classifier.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/classifier/df/tools/FrequenciesJob.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/classifier/naivebayes/training/TrainNaiveBayesJob.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/spectral/common/AffinityMatrixInputJob.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/spectral/common/MatrixDiagonalizeJob.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/spectral/common/UnitVectorizerJob.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/spectral/common/VectorMatrixMultiplicationJob.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/spectral/eigencuts/EigencutsAffinityCutsJob.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/clustering/spectral/eigencuts/EigencutsSensitivityJob.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/fpm/pfpgrowth/PFPGrowth.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/ga/watchmaker/MahoutEvaluator.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/graph/AdjacencyMatrixJob.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/graph/linkanalysis/RandomWalk.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/math/hadoop/similarity/cooccurrence/RowSimilarityJob.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/math/hadoop/stats/BasicStats.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/math/stats/entropy/ConditionalEntropy.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/math/stats/entropy/Entropy.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/vectorizer/DictionaryVectorizer.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/vectorizer/DocumentProcessor.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/vectorizer/HighDFWordsPruner.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/vectorizer/SimpleTextEncodingVectorizer.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/vectorizer/collocations/llr/CollocDriver.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/vectorizer/common/PartialVectorMerger.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/vectorizer/tfidf/TFIDFConverter.java
* /mahout/trunk/examples/src/main/java/org/apache/mahout/cf/taste/example/email/MailToPrefsDriver.java
* /mahout/trunk/examples/src/main/java/org/apache/mahout/classifier/bayes/WikipediaDatasetCreatorDriver.java
* /mahout/trunk/examples/src/main/java/org/apache/mahout/classifier/email/PrepEmailVectorsDriver.java
* /mahout/trunk/examples/src/main/java/org/apache/mahout/fpm/pfpgrowth/dataset/KeyBasedStringTupleGrouper.java
* /mahout/trunk/examples/src/main/java/org/apache/mahout/ga/watchmaker/cd/hadoop/CDMahoutEvaluator.java
* /mahout/trunk/examples/src/main/java/org/apache/mahout/ga/watchmaker/cd/tool/CDInfosTool.java
* /mahout/trunk/examples/src/main/java/org/apache/mahout/text/WikipediaToSequenceFile.java
* /mahout/trunk/integration/src/main/java/org/apache/mahout/clustering/conversion/InputDriver.java
* /mahout/trunk/integration/src/main/java/org/apache/mahout/clustering/conversion/meanshift/InputDriver.java
* /mahout/trunk/integration/src/main/java/org/apache/mahout/clustering/evaluation/RepresentativePointsDriver.java
* /mahout/trunk/integration/src/main/java/org/apache/mahout/text/TextParagraphSplittingJob.java
* /mahout/trunk/integration/src/main/java/org/apache/mahout/utils/SplitInputJob.java
* /mahout/trunk/integration/src/main/java/org/apache/mahout/utils/regex/RegexConverterDriver.java
> Map-reduce job status often left unchecked
> ------------------------------------------
>
> Key: MAHOUT-946
> URL: https://issues.apache.org/jira/browse/MAHOUT-946
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.6
> Reporter: tom pierce
> Assignee: Sean Owen
> Fix For: 0.7
>
> Attachments: MAHOUT-946.patch, MAHOUT-946.patch
>
>
> I've run into a few places in Mahout where mapreduce jobs can fail and their status won't be checked, so processing will continue on. This sometimes obscures the root problem. I've tracked down a bunch of places where this problem exists, and tried to fix the code to do something reasonable (return -1 status code, throw exception, etc.) when jobs fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-946) Map-reduce job status often left
unchecked
Posted by "Lance Norskog (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186626#comment-13186626 ]
Lance Norskog commented on MAHOUT-946:
--------------------------------------
Don't forget the example shell scripts!
> Map-reduce job status often left unchecked
> ------------------------------------------
>
> Key: MAHOUT-946
> URL: https://issues.apache.org/jira/browse/MAHOUT-946
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.6
> Reporter: tom pierce
> Attachments: MAHOUT-946.patch
>
>
> I've run into a few places in Mahout where mapreduce jobs can fail and their status won't be checked, so processing will continue on. This sometimes obscures the root problem. I've tracked down a bunch of places where this problem exists, and tried to fix the code to do something reasonable (return -1 status code, throw exception, etc.) when jobs fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-946) Map-reduce job status often left
unchecked
Posted by "Dmitriy Lyubimov (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202917#comment-13202917 ]
Dmitriy Lyubimov commented on MAHOUT-946:
-----------------------------------------
bq. Some of these classes are also not AbstractJobs (though in some cases they should probably be converted - not sure it would make sense in all cases), some return some non-status value from the method that fires of the MR job, etc. Maybe a static util method would make more sense?
Static util method would probably make more sense.
My current view of AbstractJob is that it mixing two disparate concerns in one implementation: a Tool (pipeline bootstrap) in a sense of handing off the default Configuration, to which end it has a lot of helper methods, and a distinct MR job. Due to this, it is hard (or at least inconsistent) to use it as a base for a separate and highly specific step in a multistep pipelines.
Static method would not require AbstractJob as a base, so static drop-in is probably a better approach at this point.
> Map-reduce job status often left unchecked
> ------------------------------------------
>
> Key: MAHOUT-946
> URL: https://issues.apache.org/jira/browse/MAHOUT-946
> Project: Mahout
> Issue Type: Bug
> Affects Versions: 0.6
> Reporter: tom pierce
> Attachments: MAHOUT-946.patch, MAHOUT-946.patch
>
>
> I've run into a few places in Mahout where mapreduce jobs can fail and their status won't be checked, so processing will continue on. This sometimes obscures the root problem. I've tracked down a bunch of places where this problem exists, and tried to fix the code to do something reasonable (return -1 status code, throw exception, etc.) when jobs fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira