You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Lance Norskog (JIRA)" <ji...@apache.org> on 2011/01/31 08:11:12 UTC
[jira] Created: (MAHOUT-602) "Partial Implementation" throws
exceptions
"Partial Implementation" throws exceptions
------------------------------------------
Key: MAHOUT-602
URL: https://issues.apache.org/jira/browse/MAHOUT-602
Project: Mahout
Issue Type: Bug
Environment: Macos X
java version "1.6.0_22"
Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
Reporter: Lance Norskog
The "Partial Implementation" described on the wiki page [Partial Implementation|https://cwiki.apache.org/confluence/display/MAHOUT/Partial+Implementation] fails with the given dataset and operations.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAHOUT-602) "Partial Implementation" throws
exceptions
Posted by "Deneche A. Hakim (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988861#comment-12988861 ]
Deneche A. Hakim commented on MAHOUT-602:
-----------------------------------------
What version of Mahout are you using ? I remember trying this example just before the release of Mahout-0.4 and it was Ok. And I just tried it using the latest checkout from svn and it worked perfectly, although I didn't use the mahout script but called hadoop direclty as shown in the wiki.
did you change the data as described in the wiki ?
{quote}
Open the train and test files and remove all the lines that begin with '@'. All those lines are at the top of the files. Actually you can keep those lines somewhere, because they'll help us describe the dataset to Mahout
{quote}
> "Partial Implementation" throws exceptions
> ------------------------------------------
>
> Key: MAHOUT-602
> URL: https://issues.apache.org/jira/browse/MAHOUT-602
> Project: Mahout
> Issue Type: Bug
> Environment: Macos X
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
> Reporter: Lance Norskog
> Attachments: partialImp_fullKDD_errors.log
>
>
> The "Partial Implementation" described on the wiki page [Partial Implementation|https://cwiki.apache.org/confluence/display/MAHOUT/Partial+Implementation] fails with the given dataset and operations.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAHOUT-602) "Partial Implementation" throws
exceptions
Posted by "Deneche A. Hakim (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990223#comment-12990223 ]
Deneche A. Hakim commented on MAHOUT-602:
-----------------------------------------
Ok I was finaly able to reproduce the bug, it happens only when Hadoop is running in 'local' (not even pseudo-distributed). I will try to post a patch as soon as possible.
But still I wasn't able to reproduce the first error (with Describe) even in 'local' mode.
> "Partial Implementation" throws exceptions
> ------------------------------------------
>
> Key: MAHOUT-602
> URL: https://issues.apache.org/jira/browse/MAHOUT-602
> Project: Mahout
> Issue Type: Bug
> Environment: Macos X
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
> Reporter: Lance Norskog
> Attachments: 10_hashCode.log, 10_toString.log, PartialImplementationBug1.patch, partialImp_fullKDD_errors.log
>
>
> The "Partial Implementation" described on the wiki page [Partial Implementation|https://cwiki.apache.org/confluence/display/MAHOUT/Partial+Implementation] fails with the given dataset and operations.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAHOUT-602) "Partial Implementation" throws
exceptions
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990024#comment-12990024 ]
Sean Owen commented on MAHOUT-602:
----------------------------------
You aren't just referring to the fact that you'll get part-r-00000, part-r-00001, etc. ? that's normal.
> "Partial Implementation" throws exceptions
> ------------------------------------------
>
> Key: MAHOUT-602
> URL: https://issues.apache.org/jira/browse/MAHOUT-602
> Project: Mahout
> Issue Type: Bug
> Environment: Macos X
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
> Reporter: Lance Norskog
> Attachments: 10_hashCode.log, 10_toString.log, PartialImplementationBug1.patch, partialImp_fullKDD_errors.log
>
>
> The "Partial Implementation" described on the wiki page [Partial Implementation|https://cwiki.apache.org/confluence/display/MAHOUT/Partial+Implementation] fails with the given dataset and operations.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAHOUT-602) "Partial Implementation" throws
exceptions
Posted by "Lance Norskog (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988673#comment-12988673 ]
Lance Norskog commented on MAHOUT-602:
--------------------------------------
Scenario:
I attempted to follow the tutorial on the wiki. The wiki page apparently predates the 'bin/mahout' shell script. I used that instead of the given bin/hadoop commands.
The command given to create a "file descriptor" did not work. It includes a string of letters which define a CSV file. The string is wrong.
{code}
bin/mahout org.apache.mahout.df.tools.Describe -p testdata/KDDTrain+.arff
-f testdata/KDDTrain+.info -d N 3 C 2 N C 4 N C 8 N 2 C 19 N L
{code}
This throws an exception. I added a '1' (one) to the beginning and that made it run. This seems to be the correct fix.
After this ran, I tried the next command:
{code}
bin/mahout org.apache.mahout.df.mapreduce.BuildForest
-Dmapred.max.split.size=1874231 -oob -d testdata/KDDTrain+_20Percent.arff
-ds testdata/KDDTrain+_20Percent.info -sl 5 -p -t 100 -o nsl-forest
{code}
This throws an ArrayIndexOutOfBounds exception. The full logfile is attached.
I tried this same job with different sizes of mapred.split.size, and got the same exception and index of '100'.
The logfile is attached as partialImp_fullKDD_errors.log.
> "Partial Implementation" throws exceptions
> ------------------------------------------
>
> Key: MAHOUT-602
> URL: https://issues.apache.org/jira/browse/MAHOUT-602
> Project: Mahout
> Issue Type: Bug
> Environment: Macos X
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
> Reporter: Lance Norskog
> Attachments: partialImp_fullKDD_errors.log
>
>
> The "Partial Implementation" described on the wiki page [Partial Implementation|https://cwiki.apache.org/confluence/display/MAHOUT/Partial+Implementation] fails with the given dataset and operations.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAHOUT-602) "Partial Implementation" throws
exceptions
Posted by "Lance Norskog (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lance Norskog updated MAHOUT-602:
---------------------------------
Attachment: PartialImplementationBug1.patch
Patch the reading code to show the counting error.
> "Partial Implementation" throws exceptions
> ------------------------------------------
>
> Key: MAHOUT-602
> URL: https://issues.apache.org/jira/browse/MAHOUT-602
> Project: Mahout
> Issue Type: Bug
> Environment: Macos X
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
> Reporter: Lance Norskog
> Attachments: 10_hashCode.log, PartialImplementationBug1.patch, partialImp_fullKDD_errors.log
>
>
> The "Partial Implementation" described on the wiki page [Partial Implementation|https://cwiki.apache.org/confluence/display/MAHOUT/Partial+Implementation] fails with the given dataset and operations.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAHOUT-602) "Partial Implementation" throws
exceptions
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990396#comment-12990396 ]
Sean Owen commented on MAHOUT-602:
----------------------------------
Well, it definitely is possible to write things correctly in that classes and Mappers and Reducers don't depend on being in the same JVM or on the same machine. If some code isn't doing that it's just a bug. I don't know if abstraction somehow solves the problem -- just proper encapsulation and such.
Usually it's the other way around in this kind of bug: works when stuff is in the same JVM, then breaks later. I wonder what the interaction is here that's causing the reverse?
I'd point a finger at any static members as first suspects.
> "Partial Implementation" throws exceptions
> ------------------------------------------
>
> Key: MAHOUT-602
> URL: https://issues.apache.org/jira/browse/MAHOUT-602
> Project: Mahout
> Issue Type: Bug
> Environment: Macos X
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
> Reporter: Lance Norskog
> Assignee: Deneche A. Hakim
> Attachments: 10_hashCode.log, 10_toString.log, PartialImplementationBug1.patch, partialImp_fullKDD_errors.log
>
>
> The "Partial Implementation" described on the wiki page [Partial Implementation|https://cwiki.apache.org/confluence/display/MAHOUT/Partial+Implementation] fails with the given dataset and operations.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAHOUT-602) "Partial Implementation" throws
exceptions
Posted by "Lance Norskog (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990295#comment-12990295 ]
Lance Norskog commented on MAHOUT-602:
--------------------------------------
bq. You aren't just referring to the fact that you'll get part-r-00000, part-r-00001, etc. ? that's normal.
Ask for 10 trees. You get part-r-00000 with 10 trees, and part-r-00001 with 10 trees.
> "Partial Implementation" throws exceptions
> ------------------------------------------
>
> Key: MAHOUT-602
> URL: https://issues.apache.org/jira/browse/MAHOUT-602
> Project: Mahout
> Issue Type: Bug
> Environment: Macos X
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
> Reporter: Lance Norskog
> Assignee: Deneche A. Hakim
> Attachments: 10_hashCode.log, 10_toString.log, PartialImplementationBug1.patch, partialImp_fullKDD_errors.log
>
>
> The "Partial Implementation" described on the wiki page [Partial Implementation|https://cwiki.apache.org/confluence/display/MAHOUT/Partial+Implementation] fails with the given dataset and operations.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (MAHOUT-602) "Partial Implementation" throws
exceptions
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved MAHOUT-602.
------------------------------
Resolution: Fixed
> "Partial Implementation" throws exceptions
> ------------------------------------------
>
> Key: MAHOUT-602
> URL: https://issues.apache.org/jira/browse/MAHOUT-602
> Project: Mahout
> Issue Type: Bug
> Components: Classification
> Affects Versions: 0.4
> Environment: Macos X
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
> Reporter: Lance Norskog
> Assignee: Deneche A. Hakim
> Fix For: 0.5
>
> Attachments: 10_hashCode.log, 10_toString.log, PartialImplementationBug1.patch, partialImp_fullKDD_errors.log
>
>
> The "Partial Implementation" described on the wiki page [Partial Implementation|https://cwiki.apache.org/confluence/display/MAHOUT/Partial+Implementation] fails with the given dataset and operations.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAHOUT-602) "Partial Implementation" throws
exceptions
Posted by "Lance Norskog (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989974#comment-12989974 ]
Lance Norskog commented on MAHOUT-602:
--------------------------------------
bq. did you change the data as described in the wiki ?
Yes. If you leave them in the program does not run at all.
> "Partial Implementation" throws exceptions
> ------------------------------------------
>
> Key: MAHOUT-602
> URL: https://issues.apache.org/jira/browse/MAHOUT-602
> Project: Mahout
> Issue Type: Bug
> Environment: Macos X
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
> Reporter: Lance Norskog
> Attachments: 10_hashCode.log, 10_toString.log, PartialImplementationBug1.patch, partialImp_fullKDD_errors.log
>
>
> The "Partial Implementation" described on the wiki page [Partial Implementation|https://cwiki.apache.org/confluence/display/MAHOUT/Partial+Implementation] fails with the given dataset and operations.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAHOUT-602) "Partial Implementation" throws
exceptions
Posted by "Lance Norskog (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lance Norskog updated MAHOUT-602:
---------------------------------
Attachment: 10_toString.log
Error log for 10 trees. Prints actual tree strings; rather less readable than the hashCode() output.
> "Partial Implementation" throws exceptions
> ------------------------------------------
>
> Key: MAHOUT-602
> URL: https://issues.apache.org/jira/browse/MAHOUT-602
> Project: Mahout
> Issue Type: Bug
> Environment: Macos X
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
> Reporter: Lance Norskog
> Attachments: 10_hashCode.log, 10_toString.log, PartialImplementationBug1.patch, partialImp_fullKDD_errors.log
>
>
> The "Partial Implementation" described on the wiki page [Partial Implementation|https://cwiki.apache.org/confluence/display/MAHOUT/Partial+Implementation] fails with the given dataset and operations.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAHOUT-602) "Partial Implementation" throws
exceptions
Posted by "Lance Norskog (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990780#comment-12990780 ]
Lance Norskog commented on MAHOUT-602:
--------------------------------------
Yes, that fixed it. Thanks!
Also, the 'data description language' string from the wiki page works now. I probably misinterpreted exceptions before.
> "Partial Implementation" throws exceptions
> ------------------------------------------
>
> Key: MAHOUT-602
> URL: https://issues.apache.org/jira/browse/MAHOUT-602
> Project: Mahout
> Issue Type: Bug
> Components: Classification
> Affects Versions: 0.4
> Environment: Macos X
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
> Reporter: Lance Norskog
> Assignee: Deneche A. Hakim
> Fix For: 0.5
>
> Attachments: 10_hashCode.log, 10_toString.log, PartialImplementationBug1.patch, partialImp_fullKDD_errors.log
>
>
> The "Partial Implementation" described on the wiki page [Partial Implementation|https://cwiki.apache.org/confluence/display/MAHOUT/Partial+Implementation] fails with the given dataset and operations.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAHOUT-602) "Partial Implementation" throws
exceptions
Posted by "Lance Norskog (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lance Norskog updated MAHOUT-602:
---------------------------------
Attachment: partialImp_fullKDD_errors.log
> "Partial Implementation" throws exceptions
> ------------------------------------------
>
> Key: MAHOUT-602
> URL: https://issues.apache.org/jira/browse/MAHOUT-602
> Project: Mahout
> Issue Type: Bug
> Environment: Macos X
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
> Reporter: Lance Norskog
> Attachments: partialImp_fullKDD_errors.log
>
>
> The "Partial Implementation" described on the wiki page [Partial Implementation|https://cwiki.apache.org/confluence/display/MAHOUT/Partial+Implementation] fails with the given dataset and operations.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (MAHOUT-602) "Partial Implementation"
throws exceptions
Posted by "Lance Norskog (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989970#comment-12989970 ]
Lance Norskog edited comment on MAHOUT-602 at 2/3/11 5:44 AM:
--------------------------------------------------------------
Error log for "-t 10" meaning "generate 10 trees". Includes prints of the hashCode() for each tree. All are different.
Since the hashCode() output is different, that means each tree is unique.
was (Author: lancenorskog):
Error log for 10 trees. Includes prints of the hashCode() for each tree. All are different.
> "Partial Implementation" throws exceptions
> ------------------------------------------
>
> Key: MAHOUT-602
> URL: https://issues.apache.org/jira/browse/MAHOUT-602
> Project: Mahout
> Issue Type: Bug
> Environment: Macos X
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
> Reporter: Lance Norskog
> Attachments: 10_hashCode.log, 10_toString.log, PartialImplementationBug1.patch, partialImp_fullKDD_errors.log
>
>
> The "Partial Implementation" described on the wiki page [Partial Implementation|https://cwiki.apache.org/confluence/display/MAHOUT/Partial+Implementation] fails with the given dataset and operations.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Assigned: (MAHOUT-602) "Partial Implementation" throws
exceptions
Posted by "Deneche A. Hakim (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Deneche A. Hakim reassigned MAHOUT-602:
---------------------------------------
Assignee: Deneche A. Hakim
> "Partial Implementation" throws exceptions
> ------------------------------------------
>
> Key: MAHOUT-602
> URL: https://issues.apache.org/jira/browse/MAHOUT-602
> Project: Mahout
> Issue Type: Bug
> Environment: Macos X
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
> Reporter: Lance Norskog
> Assignee: Deneche A. Hakim
> Attachments: 10_hashCode.log, 10_toString.log, PartialImplementationBug1.patch, partialImp_fullKDD_errors.log
>
>
> The "Partial Implementation" described on the wiki page [Partial Implementation|https://cwiki.apache.org/confluence/display/MAHOUT/Partial+Implementation] fails with the given dataset and operations.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAHOUT-602) "Partial Implementation" throws
exceptions
Posted by "Ted Dunning (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990374#comment-12990374 ]
Ted Dunning commented on MAHOUT-602:
------------------------------------
{quote}
Is it possible to abstract out all such problems so that the Hadoop mode does not affect Mahout code?
{quote}
It is sort-of possible.
The big problem is that in local mode, everything runs in the same JVM. That has lots of implications about the accidental use of shared memory. Many a Hadoop beginner has been very confused when their program fails dramatically when first run on a truly distributed model.
> "Partial Implementation" throws exceptions
> ------------------------------------------
>
> Key: MAHOUT-602
> URL: https://issues.apache.org/jira/browse/MAHOUT-602
> Project: Mahout
> Issue Type: Bug
> Environment: Macos X
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
> Reporter: Lance Norskog
> Assignee: Deneche A. Hakim
> Attachments: 10_hashCode.log, 10_toString.log, PartialImplementationBug1.patch, partialImp_fullKDD_errors.log
>
>
> The "Partial Implementation" described on the wiki page [Partial Implementation|https://cwiki.apache.org/confluence/display/MAHOUT/Partial+Implementation] fails with the given dataset and operations.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAHOUT-602) "Partial Implementation" throws
exceptions
Posted by "Lance Norskog (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989968#comment-12989968 ]
Lance Norskog commented on MAHOUT-602:
--------------------------------------
There is an off-by-one error somewhere. The code generates two files with 'number of trees requested' instead of one.
To make things easier I created 10 trees instead of 100. Two files of trees are created instead of just one. The patch prints the hashCode() for each tree.toString. You can see that the two files have different trees. I have included the value for each tree in the attached log 10_hashCode.log. (10_toString.log shows the actual string dump for each tree.)
Apply the patch attached as PartialImplementationBug1.patch, if you want to recreate the experiment. Try different numbers of trees and it will always make two files of N trees instead of just 1.
This was the command line, as per the wiki:
$HADOOP_HOME/bin/hadoop jar /Users/lancenorskog/Documents/open/mahout/examples/target/mahout-examples-0.5-SNAPSHOT-job.jar org.apache.mahout.df.mapreduce.BuildForest -Dmapred.max.split.size=1874231 -oob -d ../../datasets/KDDTrain/KDDTrain+_20Percent.arff -ds ../../datasets/KDDTrain/KDDTrain+_20Percent.info -sl 5 -p -t 10 -o nsl-forest
> "Partial Implementation" throws exceptions
> ------------------------------------------
>
> Key: MAHOUT-602
> URL: https://issues.apache.org/jira/browse/MAHOUT-602
> Project: Mahout
> Issue Type: Bug
> Environment: Macos X
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
> Reporter: Lance Norskog
> Attachments: partialImp_fullKDD_errors.log
>
>
> The "Partial Implementation" described on the wiki page [Partial Implementation|https://cwiki.apache.org/confluence/display/MAHOUT/Partial+Implementation] fails with the given dataset and operations.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAHOUT-602) "Partial Implementation" throws
exceptions
Posted by "Lance Norskog (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lance Norskog updated MAHOUT-602:
---------------------------------
Attachment: 10_hashCode.log
Error log for 10 trees. Includes prints of the hashCode() for each tree. All are different.
> "Partial Implementation" throws exceptions
> ------------------------------------------
>
> Key: MAHOUT-602
> URL: https://issues.apache.org/jira/browse/MAHOUT-602
> Project: Mahout
> Issue Type: Bug
> Environment: Macos X
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
> Reporter: Lance Norskog
> Attachments: 10_hashCode.log, PartialImplementationBug1.patch, partialImp_fullKDD_errors.log
>
>
> The "Partial Implementation" described on the wiki page [Partial Implementation|https://cwiki.apache.org/confluence/display/MAHOUT/Partial+Implementation] fails with the given dataset and operations.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAHOUT-602) "Partial Implementation" throws
exceptions
Posted by "Lance Norskog (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990296#comment-12990296 ]
Lance Norskog commented on MAHOUT-602:
--------------------------------------
bq. Ok I was finaly able to reproduce the bug, it happens only when Hadoop is running in 'local' (not even pseudo-distributed). I will try to post a patch as soon as possible.
Thank you. I only use local mode so it didn't occur to try it in the distributed modes. Is it possible to abstract out all such problems so that the Hadoop mode does not affect Mahout code?
> "Partial Implementation" throws exceptions
> ------------------------------------------
>
> Key: MAHOUT-602
> URL: https://issues.apache.org/jira/browse/MAHOUT-602
> Project: Mahout
> Issue Type: Bug
> Environment: Macos X
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
> Reporter: Lance Norskog
> Assignee: Deneche A. Hakim
> Attachments: 10_hashCode.log, 10_toString.log, PartialImplementationBug1.patch, partialImp_fullKDD_errors.log
>
>
> The "Partial Implementation" described on the wiki page [Partial Implementation|https://cwiki.apache.org/confluence/display/MAHOUT/Partial+Implementation] fails with the given dataset and operations.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAHOUT-602) "Partial Implementation" throws
exceptions
Posted by "Deneche A. Hakim (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990465#comment-12990465 ]
Deneche A. Hakim commented on MAHOUT-602:
-----------------------------------------
When I first wrote this code (two years ago) I found that when running in 'local' mode Hadoop doesn't launch more than one mapper even if the partition size should create many. And because it's important for the code to know how many mappers did run in the first step, I made the code check if it's in 'local' mode and assume that only 1 single mapper did run in that case. But it looks like that now, even in 'local' mode, many mappers are actually launched.
> "Partial Implementation" throws exceptions
> ------------------------------------------
>
> Key: MAHOUT-602
> URL: https://issues.apache.org/jira/browse/MAHOUT-602
> Project: Mahout
> Issue Type: Bug
> Environment: Macos X
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
> Reporter: Lance Norskog
> Assignee: Deneche A. Hakim
> Attachments: 10_hashCode.log, 10_toString.log, PartialImplementationBug1.patch, partialImp_fullKDD_errors.log
>
>
> The "Partial Implementation" described on the wiki page [Partial Implementation|https://cwiki.apache.org/confluence/display/MAHOUT/Partial+Implementation] fails with the given dataset and operations.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAHOUT-602) "Partial Implementation" throws
exceptions
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990498#comment-12990498 ]
Hudson commented on MAHOUT-602:
-------------------------------
Integrated in Mahout-Quality #609 (See [https://hudson.apache.org/hudson/job/Mahout-Quality/609/])
MAHOUT-602 Removed unnecessary code and comment
MAHOUT-602 Removed the check for Hadoop local mode
> "Partial Implementation" throws exceptions
> ------------------------------------------
>
> Key: MAHOUT-602
> URL: https://issues.apache.org/jira/browse/MAHOUT-602
> Project: Mahout
> Issue Type: Bug
> Environment: Macos X
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
> Reporter: Lance Norskog
> Assignee: Deneche A. Hakim
> Attachments: 10_hashCode.log, 10_toString.log, PartialImplementationBug1.patch, partialImp_fullKDD_errors.log
>
>
> The "Partial Implementation" described on the wiki page [Partial Implementation|https://cwiki.apache.org/confluence/display/MAHOUT/Partial+Implementation] fails with the given dataset and operations.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAHOUT-602) "Partial Implementation" throws
exceptions
Posted by "Sean Owen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen updated MAHOUT-602:
-----------------------------
Component/s: Classification
Due Date: 11/Feb/11
Affects Version/s: 0.4
Fix Version/s: 0.5
Lance, does the latest commit fix the issue?
> "Partial Implementation" throws exceptions
> ------------------------------------------
>
> Key: MAHOUT-602
> URL: https://issues.apache.org/jira/browse/MAHOUT-602
> Project: Mahout
> Issue Type: Bug
> Components: Classification
> Affects Versions: 0.4
> Environment: Macos X
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)
> Reporter: Lance Norskog
> Assignee: Deneche A. Hakim
> Fix For: 0.5
>
> Attachments: 10_hashCode.log, 10_toString.log, PartialImplementationBug1.patch, partialImp_fullKDD_errors.log
>
>
> The "Partial Implementation" described on the wiki page [Partial Implementation|https://cwiki.apache.org/confluence/display/MAHOUT/Partial+Implementation] fails with the given dataset and operations.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira