You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2011/05/21 11:54:47 UTC
[jira] [Created] (MAHOUT-708) Update to Hadoop 0.21
Update to Hadoop 0.21
---------------------
Key: MAHOUT-708
URL: https://issues.apache.org/jira/browse/MAHOUT-708
Project: Mahout
Issue Type: Task
Components: Classification, Clustering, Collaborative Filtering, Frequent Itemset/Association Rule Mining
Affects Versions: 0.5
Reporter: Sean Owen
Assignee: Sean Owen
Fix For: 0.6
I suggest we should move to Hadoop 0.21 for the next release. It is the current release, soon to be superseded by 0.22. It matches more closely what CDH3/4 users use. It has bug fixes, and crucially some features that make joins much less painful.
The drawback is that EMR does not yet support it. I still suggest we forge ahead as one imagines it will be supported by the time we release 0.6.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-708) Update to Hadoop 0.21
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042691#comment-13042691 ]
Hudson commented on MAHOUT-708:
-------------------------------
Integrated in Mahout-Quality #848 (See [https://builds.apache.org/hudson/job/Mahout-Quality/848/])
> Update to Hadoop 0.21
> ---------------------
>
> Key: MAHOUT-708
> URL: https://issues.apache.org/jira/browse/MAHOUT-708
> Project: Mahout
> Issue Type: Task
> Components: Classification, Clustering, Collaborative Filtering, Frequent Itemset/Association Rule Mining
> Affects Versions: 0.5, 0.6
> Reporter: Sean Owen
> Assignee: Sean Owen
> Labels: hadoop
> Fix For: 1.0
>
>
> I suggest we should move to Hadoop 0.21 for the next release. It is the current release, soon to be superseded by 0.22. It matches more closely what CDH3/4 users use. It has bug fixes, and crucially some features that make joins much less painful.
> The drawback is that EMR does not yet support it. I still suggest we forge ahead as one imagines it will be supported by the time we release 0.6.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-708) Update to Hadoop 0.20.203.0
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13054197#comment-13054197 ]
Hudson commented on MAHOUT-708:
-------------------------------
Integrated in Mahout-Quality #901 (See [https://builds.apache.org/job/Mahout-Quality/901/])
MAHOUT-708 update to Hadoop 0.20.203.0, which just entailed better logic to ignore new _SUCCESS files. The result still works in 0.20.2
srowen : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1139072
Files :
* /mahout/trunk/core/src/main/java/org/apache/mahout/common/iterator/sequencefile/SequenceFileDirIterable.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/ga/watchmaker/OutputUtils.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/math/hadoop/DistributedRowMatrix.java
* /mahout/trunk/pom.xml
* /mahout/trunk/core/src/main/java/org/apache/mahout/common/iterator/sequencefile/SequenceFileDirIterator.java
* /mahout/trunk/core/pom.xml
* /mahout/trunk/core/src/main/java/org/apache/mahout/common/iterator/sequencefile/SequenceFileDirValueIterable.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/cf/taste/hadoop/als/eval/ParallelFactorizationEvaluator.java
* /mahout/trunk/core/src/test/java/org/apache/mahout/math/hadoop/TestDistributedRowMatrix.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/common/iterator/sequencefile/SequenceFileDirValueIterator.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/common/iterator/sequencefile/SequenceFileIterable.java
* /mahout/trunk/core/src/test/java/org/apache/mahout/vectorizer/DocumentProcessorTest.java
* /mahout/trunk/core/src/main/java/org/apache/mahout/common/iterator/sequencefile/SequenceFileValueIterable.java
> Update to Hadoop 0.20.203.0
> ---------------------------
>
> Key: MAHOUT-708
> URL: https://issues.apache.org/jira/browse/MAHOUT-708
> Project: Mahout
> Issue Type: Task
> Components: Classification, Clustering, Collaborative Filtering, Frequent Itemset/Association Rule Mining
> Affects Versions: 0.5
> Reporter: Sean Owen
> Assignee: Sean Owen
> Labels: hadoop
> Fix For: 0.6
>
>
> I suggest we should move to Hadoop 0.20.203.0 for the next release. (Not 0.21 or further.) It is a much more recent branch of 0.20.x and is compile-time compatible with 0.20.2 in our code already.
> However I know already that switching to it causes some failures, in the Lanczos jobs for instances. Looks like something's expecting a file somewhere that isn't where it used to be. I bet it's an easy fix, but don't know what it is yet.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAHOUT-708) Update to Hadoop 0.20.203.0
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved MAHOUT-708.
------------------------------
Resolution: Fixed
So, updating to 0.20.203.0 was almost painless. There were two problems.
Hadoop 0.20.203.0 depends on Jackson from Codehaus, but doesn't declare it in the POM. So we had to add that dependency manually.
And, it also writes _SUCCESS files in output dirs. Many bits of code and tests still didn't correctly filter these out. this was easy to fix. Incidentally this ought to fix some problems people see on CDH, which has the same behavior.
I should stress that the resulting code is still entirely compatible with Hadoop 0.20.2. We haven't really upped requirements.
> Update to Hadoop 0.20.203.0
> ---------------------------
>
> Key: MAHOUT-708
> URL: https://issues.apache.org/jira/browse/MAHOUT-708
> Project: Mahout
> Issue Type: Task
> Components: Classification, Clustering, Collaborative Filtering, Frequent Itemset/Association Rule Mining
> Affects Versions: 0.5
> Reporter: Sean Owen
> Assignee: Sean Owen
> Labels: hadoop
> Fix For: 0.6
>
>
> I suggest we should move to Hadoop 0.20.203.0 for the next release. (Not 0.21 or further.) It is a much more recent branch of 0.20.x and is compile-time compatible with 0.20.2 in our code already.
> However I know already that switching to it causes some failures, in the Lanczos jobs for instances. Looks like something's expecting a file somewhere that isn't where it used to be. I bet it's an easy fix, but don't know what it is yet.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-708) Update to Hadoop 0.21
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated MAHOUT-708:
-----------------------------
Affects Version/s: 0.6
Fix Version/s: (was: 0.6)
1.0
> Update to Hadoop 0.21
> ---------------------
>
> Key: MAHOUT-708
> URL: https://issues.apache.org/jira/browse/MAHOUT-708
> Project: Mahout
> Issue Type: Task
> Components: Classification, Clustering, Collaborative Filtering, Frequent Itemset/Association Rule Mining
> Affects Versions: 0.5, 0.6
> Reporter: Sean Owen
> Assignee: Sean Owen
> Labels: hadoop
> Fix For: 1.0
>
>
> I suggest we should move to Hadoop 0.21 for the next release. It is the current release, soon to be superseded by 0.22. It matches more closely what CDH3/4 users use. It has bug fixes, and crucially some features that make joins much less painful.
> The drawback is that EMR does not yet support it. I still suggest we forge ahead as one imagines it will be supported by the time we release 0.6.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-708) Update to Hadoop 0.20.203.0
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated MAHOUT-708:
-----------------------------
Description:
I suggest we should move to Hadoop 0.20.203.0 for the next release. (Not 0.21 or further.) It is a much more recent branch of 0.20.x and is compile-time compatible with 0.20.2 in our code already.
However I know already that switching to it causes some failures, in the Lanczos jobs for instances. Looks like something's expecting a file somewhere that isn't where it used to be. I bet it's an easy fix, but don't know what it is yet.
was:
I suggest we should move to Hadoop 0.21 for the next release. It is the current release, soon to be superseded by 0.22. It matches more closely what CDH3/4 users use. It has bug fixes, and crucially some features that make joins much less painful.
The drawback is that EMR does not yet support it. I still suggest we forge ahead as one imagines it will be supported by the time we release 0.6.
Affects Version/s: (was: 0.6)
Fix Version/s: (was: 1.0)
0.6
Summary: Update to Hadoop 0.20.203.0 (was: Update to Hadoop 0.21)
> Update to Hadoop 0.20.203.0
> ---------------------------
>
> Key: MAHOUT-708
> URL: https://issues.apache.org/jira/browse/MAHOUT-708
> Project: Mahout
> Issue Type: Task
> Components: Classification, Clustering, Collaborative Filtering, Frequent Itemset/Association Rule Mining
> Affects Versions: 0.5
> Reporter: Sean Owen
> Assignee: Sean Owen
> Labels: hadoop
> Fix For: 0.6
>
>
> I suggest we should move to Hadoop 0.20.203.0 for the next release. (Not 0.21 or further.) It is a much more recent branch of 0.20.x and is compile-time compatible with 0.20.2 in our code already.
> However I know already that switching to it causes some failures, in the Lanczos jobs for instances. Looks like something's expecting a file somewhere that isn't where it used to be. I bet it's an easy fix, but don't know what it is yet.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-708) Update to Hadoop 0.21
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037462#comment-13037462 ]
Sean Owen commented on MAHOUT-708:
----------------------------------
Let's sit on this a while longer then. We should at least get onto 0.20.203.
Yes, it brings back joins and multiple outputs, which is 80% of the reason we'd want it.
I'm on 0.22 and it makes it possible to build a recommender pipeline half as complex and about 5x as fast. It's big for some machine learning apps.
> Update to Hadoop 0.21
> ---------------------
>
> Key: MAHOUT-708
> URL: https://issues.apache.org/jira/browse/MAHOUT-708
> Project: Mahout
> Issue Type: Task
> Components: Classification, Clustering, Collaborative Filtering, Frequent Itemset/Association Rule Mining
> Affects Versions: 0.5
> Reporter: Sean Owen
> Assignee: Sean Owen
> Labels: hadoop
> Fix For: 0.6
>
>
> I suggest we should move to Hadoop 0.21 for the next release. It is the current release, soon to be superseded by 0.22. It matches more closely what CDH3/4 users use. It has bug fixes, and crucially some features that make joins much less painful.
> The drawback is that EMR does not yet support it. I still suggest we forge ahead as one imagines it will be supported by the time we release 0.6.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-708) Update to Hadoop 0.21
Posted by "Jake Mannix (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037461#comment-13037461 ]
Jake Mannix commented on MAHOUT-708:
------------------------------------
I'm open to being convinced, but as stated on the list, I'm starting from a position of -1 on this.
For the same reason that Lucene didn't move to Java 5 until *last year*, we may be stuck with mixed 0.18/0.20+ apis for quite some time.
Anyone running their own cluster can certainly upgrade to 0.22, but production systems, by and large, are far more conservative, and have also often been bitten by hadoop upgrades (0.19, anyone?) in the past, and so are really loathe to just jump onto the newest "blessed" release.
If someone can convince me that everyone has moved on to 0.21+, then sure. But so far, that is not my experience, at all.
> Update to Hadoop 0.21
> ---------------------
>
> Key: MAHOUT-708
> URL: https://issues.apache.org/jira/browse/MAHOUT-708
> Project: Mahout
> Issue Type: Task
> Components: Classification, Clustering, Collaborative Filtering, Frequent Itemset/Association Rule Mining
> Affects Versions: 0.5
> Reporter: Sean Owen
> Assignee: Sean Owen
> Labels: hadoop
> Fix For: 0.6
>
>
> I suggest we should move to Hadoop 0.21 for the next release. It is the current release, soon to be superseded by 0.22. It matches more closely what CDH3/4 users use. It has bug fixes, and crucially some features that make joins much less painful.
> The drawback is that EMR does not yet support it. I still suggest we forge ahead as one imagines it will be supported by the time we release 0.6.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-708) Update to Hadoop 0.21
Posted by "Ted Dunning (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037463#comment-13037463 ]
Ted Dunning commented on MAHOUT-708:
------------------------------------
Frankly, I think that the next version of Hadoop that provides any compelling features for large enterprises is likely to be 0.23 where MR nextgen comes into play. Ironically, the major reason that version will be exciting is that it allows for compatibility with old API's. The ability to have old and new side-by-side is a pre-requisite for any large cluster upgrade.
> Update to Hadoop 0.21
> ---------------------
>
> Key: MAHOUT-708
> URL: https://issues.apache.org/jira/browse/MAHOUT-708
> Project: Mahout
> Issue Type: Task
> Components: Classification, Clustering, Collaborative Filtering, Frequent Itemset/Association Rule Mining
> Affects Versions: 0.5
> Reporter: Sean Owen
> Assignee: Sean Owen
> Labels: hadoop
> Fix For: 0.6
>
>
> I suggest we should move to Hadoop 0.21 for the next release. It is the current release, soon to be superseded by 0.22. It matches more closely what CDH3/4 users use. It has bug fixes, and crucially some features that make joins much less painful.
> The drawback is that EMR does not yet support it. I still suggest we forge ahead as one imagines it will be supported by the time we release 0.6.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-708) Update to Hadoop 0.21
Posted by "Dmitriy Lyubimov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037699#comment-13037699 ]
Dmitriy Lyubimov commented on MAHOUT-708:
-----------------------------------------
-1 in general.
Most folks are either on 0.20.2 (EMR) or CDH3 (baremetal). I know of no one using 0.21. I am not sure that using 0.21 new api will be 100% compatible with CDH3, there are still some missing pieces there. So if you move, you may have me locked in 0.5 since i am a CDH3 user. (and EMR for bigger trains).
What i think might be reasonable is to create a branch with cdh3 dependencies and make sure all tests are passing (i saw 2 or 3 not passing), albeit generally everything compiles with cdh3. _Then we would cover all major camps out there_ with practically same codebase.
Yes i am also waiting for new hadoop architecture to come out (i think they were saying mid summer), a fundamental rewrite where task resource is separated from a concept of application (i.e. map reduce) and that would really be great. That would be a worthy update.
-d
> Update to Hadoop 0.21
> ---------------------
>
> Key: MAHOUT-708
> URL: https://issues.apache.org/jira/browse/MAHOUT-708
> Project: Mahout
> Issue Type: Task
> Components: Classification, Clustering, Collaborative Filtering, Frequent Itemset/Association Rule Mining
> Affects Versions: 0.5, 0.6
> Reporter: Sean Owen
> Assignee: Sean Owen
> Labels: hadoop
> Fix For: 1.0
>
>
> I suggest we should move to Hadoop 0.21 for the next release. It is the current release, soon to be superseded by 0.22. It matches more closely what CDH3/4 users use. It has bug fixes, and crucially some features that make joins much less painful.
> The drawback is that EMR does not yet support it. I still suggest we forge ahead as one imagines it will be supported by the time we release 0.6.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-708) Update to Hadoop 0.21
Posted by "Shannon Quinn (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037449#comment-13037449 ]
Shannon Quinn commented on MAHOUT-708:
--------------------------------------
Just like this! (re: my last email)
> Update to Hadoop 0.21
> ---------------------
>
> Key: MAHOUT-708
> URL: https://issues.apache.org/jira/browse/MAHOUT-708
> Project: Mahout
> Issue Type: Task
> Components: Classification, Clustering, Collaborative Filtering, Frequent Itemset/Association Rule Mining
> Affects Versions: 0.5
> Reporter: Sean Owen
> Assignee: Sean Owen
> Labels: hadoop
> Fix For: 0.6
>
>
> I suggest we should move to Hadoop 0.21 for the next release. It is the current release, soon to be superseded by 0.22. It matches more closely what CDH3/4 users use. It has bug fixes, and crucially some features that make joins much less painful.
> The drawback is that EMR does not yet support it. I still suggest we forge ahead as one imagines it will be supported by the time we release 0.6.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira