You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Stuti Awasthi <st...@hcl.com> on 2013/01/21 13:52:00 UTC

Mahout-0.7 BuildForest execution error

Hi,

I have downloaded Mahout and tried to execute Partial Implementation. When I try to run I am getting the parsing error:

$HADOOP_HOME/hadoop jar $MAHOUT_HOME/examples/target/mahout-examples-0.7-job.jar org.apache.mahout.classifier.df.mapreduce.BuildForest -oob -d /testdata/KDDTrain+.arff -ds /testdata/KDDTrain+.info -sl 5 -p -t 100 -o /testdata/nsl-forest

13/01/21 18:16:24 ERROR mapreduce.BuildForest: Exception
org.apache.commons.cli2.OptionException: Unexpected /testdata/nsl-forest while processing Options
        at org.apache.commons.cli2.commandline.Parser.parse(Parser.java:99)
        at org.apache.mahout.classifier.df.mapreduce.BuildForest.run(BuildForest.java:139)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.mahout.classifier.df.mapreduce.BuildForest.main(BuildForest.java:253)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Usage:
[--data <path> --dataset <dataset> --selection <m> --no-complete --minsplit
<minsplit> --minprop <minprop> --seed <seed> --partial --nbtrees <nbtrees>
--output <path> --help]
Options
  --data (-d) path             Data path
  --dataset (-ds) dataset      Dataset path
  --selection (-sl) m          Optional, Number of variables to select randomly
                               at each tree-node.
                               For classification problem, the default is
                               square root of the number of explanatory
                               variables.
                               For regression problem, the default is 1/3 of
                               the number of explanatory variables.
  --no-complete (-nc)          Optional, The tree is not complemented
  --minsplit (-ms) minsplit    Optional, The tree-node is not divided, if the
                               branching data size is smaller than this value.
                               The default is 2.
  --minprop (-mp) minprop      Optional, The tree-node is not divided, if the
                               proportion of the variance of branching data is
                               smaller than this value.
                               In the case of a regression problem, this value
                               is used. The default is 1/1000(0.001).
  --seed (-sd) seed            Optional, seed value used to initialise the
                               Random number generator
  --partial (-p)               Optional, use the Partial Data implementation
  --nbtrees (-t) nbtrees       Number of trees to grow
  --output (-o) path           Output path, will contain the Decision Forest
  --help (-h)                  Print out help


If I try to run with Mahout-0.5, its working fine and generating /testdata/nsl-forest/forest.seq in hdfs.
Is this a bug in Mahout-0.7 or am I doing something wrong.

Please Suggest

Thanks
Stuti Awasthi





::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.

----------------------------------------------------------------------------------------------------------------------------------------------------

Re: Mahout-0.7 BuildForest execution error

Posted by Prasanna <pd...@gmail.com>.
Lorenz Knies <lorenz.knies <at> metrigo.de> writes:

> 
> i think the oob option is gone
> 
> On Jan 21, 2013, at 1:52 PM, Stuti Awasthi <stutiawasthi <at> hcl.com> 
wrote:
> 
> > Hi,
> > 
> > I have downloaded Mahout and tried to execute Partial Implementation. 
When I try to run I am getting the
> parsing error:
> > 
> > $HADOOP_HOME/hadoop jar $MAHOUT_HOME/examples/target/mahout-examples-
0.7-job.jar
> org.apache.mahout.classifier.df.mapreduce.BuildForest -oob -d 
/testdata/KDDTrain+.arff -ds
> /testdata/KDDTrain+.info -sl 5 -p -t 100 -o /testdata/nsl-forest
> > 
> > 13/01/21 18:16:24 ERROR mapreduce.BuildForest: Exception
> > org.apache.commons.cli2.OptionException: Unexpected /testdata/nsl-forest 
while processing Options
> >        at 
org.apache.commons.cli2.commandline.Parser.parse(Parser.java:99)
> >        at 
org.apache.mahout.classifier.df.mapreduce.BuildForest.run(BuildForest.java:1
39)
> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> >        at 
org.apache.mahout.classifier.df.mapreduce.BuildForest.main(BuildForest.java:
253)
> >        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57
)
> >        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)
> >        at java.lang.reflect.Method.invoke(Method.java:616)
> >        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> > Usage:
> > [--data <path> --dataset <dataset> --selection <m> --no-complete --
minsplit
> > <minsplit> --minprop <minprop> --seed <seed> --partial --nbtrees 
<nbtrees>
> > --output <path> --help]
> > Options
> >  --data (-d) path             Data path
> >  --dataset (-ds) dataset      Dataset path
> >  --selection (-sl) m          Optional, Number of variables to select 
randomly
> >                               at each tree-node.
> >                               For classification problem, the default is
> >                               square root of the number of explanatory
> >                               variables.
> >                               For regression problem, the default is 1/3 
of
> >                               the number of explanatory variables.
> >  --no-complete (-nc)          Optional, The tree is not complemented
> >  --minsplit (-ms) minsplit    Optional, The tree-node is not divided, if 
the
> >                               branching data size is smaller than this 
value.
> >                               The default is 2.
> >  --minprop (-mp) minprop      Optional, The tree-node is not divided, if 
the
> >                               proportion of the variance of branching 
data is
> >                               smaller than this value.
> >                               In the case of a regression problem, this 
value
> >                               is used. The default is 1/1000(0.001).
> >  --seed (-sd) seed            Optional, seed value used to initialise 
the
> >                               Random number generator
> >  --partial (-p)               Optional, use the Partial Data 
implementation
> >  --nbtrees (-t) nbtrees       Number of trees to grow
> >  --output (-o) path           Output path, will contain the Decision 
Forest
> >  --help (-h)                  Print out help
> > 
> > 
> > If I try to run with Mahout-0.5, its working fine and generating 
/testdata/nsl-forest/forest.seq in hdfs.
> > Is this a bug in Mahout-0.7 or am I doing something wrong.
> > 
> > Please Suggest
> > 
> > Thanks
> > Stuti Awasthi
> > 
> > 
> > 
> > 
> > 
> > ::DISCLAIMER::
> > ------------------------------------------------------------------------
----------------------------------------------------------------------------
> > 
> > The contents of this e-mail and any attachment(s) are confidential and 
intended for the named
> recipient(s) only.
> > E-mail transmission is not guaranteed to be secure or error-free as 
information could be intercepted, corrupted,
> > lost, destroyed, arrive late or incomplete, or may contain viruses in 
transmission. The e mail and its contents
> > (with or without referred errors) shall therefore not attach any 
liability on the originator or HCL or its affiliates.
> > Views or opinions, if any, presented in this email are solely those of 
the author and may not necessarily
> reflect the
> > views or opinions of HCL or its affiliates. Any form of reproduction, 
dissemination, copying,
> disclosure, modification,
> > distribution and / or publication of this message without the prior 
written consent of authorized
> representative of
> > HCL is strictly prohibited. If you have received this email in error 
please delete it and notify the sender immediately.
> > Before opening any email and/or attachments, please check them for 
viruses and other defects.
> > 
> > ------------------------------------------------------------------------
----------------------------------------------------------------------------
> 
> 




So how can I calculate the oob error rate in mahout 0.9?
Does the code support it? If yes what arguments do I need to give/changes do 
I need to make?
Also which version of mahout supports -oob option?


Follwing is the trace of my execution of the example:

hduser@ubuntu:/home/prasanna/Downloads/mahout-distribution-0.9$ hadoop jar 
mahout-examples-0.9-job.jar 
org.apache.mahout.classifier.df.mapreduce.BuildForest -
Dmapred.max.split.size=1874231 -d testdata/KDDTrain+.arff -ds 
testdata/KDDTrain+.info -sl 5 -p -t 10 -o nsl-forest
Warning: $HADOOP_HOME is deprecated.

14/07/10 02:17:43 INFO mapreduce.BuildForest: Partial Mapred implementation
14/07/10 02:17:43 INFO mapreduce.BuildForest: Building the forest...
14/07/10 02:17:44 INFO input.FileInputFormat: Total input paths to process : 
1
14/07/10 02:17:44 INFO util.NativeCodeLoader: Loaded the native-hadoop 
library
14/07/10 02:17:44 WARN snappy.LoadSnappy: Snappy native library not loaded
14/07/10 02:17:44 INFO mapred.JobClient: Running job: job_201407092351_0003
14/07/10 02:17:45 INFO mapred.JobClient:  map 0% reduce 0%
14/07/10 02:18:07 INFO mapred.JobClient:  map 20% reduce 0%
14/07/10 02:18:19 INFO mapred.JobClient:  map 30% reduce 0%
14/07/10 02:18:22 INFO mapred.JobClient:  map 40% reduce 0%
14/07/10 02:18:31 INFO mapred.JobClient:  map 60% reduce 0%
14/07/10 02:18:43 INFO mapred.JobClient:  map 80% reduce 0%
14/07/10 02:18:55 INFO mapred.JobClient:  map 90% reduce 0%
14/07/10 02:18:58 INFO mapred.JobClient:  map 100% reduce 0%
14/07/10 02:19:03 INFO mapred.JobClient: Job complete: job_201407092351_0003
14/07/10 02:19:03 INFO mapred.JobClient: Counters: 20
14/07/10 02:19:03 INFO mapred.JobClient:   Job Counters 
14/07/10 02:19:03 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=129677
14/07/10 02:19:03 INFO mapred.JobClient:     Total time spent by all reduces 
waiting after reserving slots (ms)=0
14/07/10 02:19:03 INFO mapred.JobClient:     Total time spent by all maps 
waiting after reserving slots (ms)=0
14/07/10 02:19:03 INFO mapred.JobClient:     Launched map tasks=10
14/07/10 02:19:03 INFO mapred.JobClient:     Data-local map tasks=10
14/07/10 02:19:03 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
14/07/10 02:19:03 INFO mapred.JobClient:   File Output Format Counters 
14/07/10 02:19:03 INFO mapred.JobClient:     Bytes Written=75424
14/07/10 02:19:03 INFO mapred.JobClient:   FileSystemCounters
14/07/10 02:19:03 INFO mapred.JobClient:     FILE_BYTES_READ=28270
14/07/10 02:19:03 INFO mapred.JobClient:     HDFS_BYTES_READ=18759170
14/07/10 02:19:03 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=227620
14/07/10 02:19:03 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=75424
14/07/10 02:19:03 INFO mapred.JobClient:   File Input Format Counters 
14/07/10 02:19:03 INFO mapred.JobClient:     Bytes Read=18757940
14/07/10 02:19:03 INFO mapred.JobClient:   Map-Reduce Framework
14/07/10 02:19:03 INFO mapred.JobClient:     Map input records=125973
14/07/10 02:19:03 INFO mapred.JobClient:     Physical memory (bytes) 
snapshot=775561216
14/07/10 02:19:03 INFO mapred.JobClient:     Spilled Records=0
14/07/10 02:19:03 INFO mapred.JobClient:     CPU time spent (ms)=20080
14/07/10 02:19:03 INFO mapred.JobClient:     Total committed heap usage 
(bytes)=317194240
14/07/10 02:19:03 INFO mapred.JobClient:     Virtual memory (bytes) 
snapshot=10654883840
14/07/10 02:19:03 INFO mapred.JobClient:     Map output records=10
14/07/10 02:19:03 INFO mapred.JobClient:     SPLIT_RAW_BYTES=1230
14/07/10 02:19:04 INFO common.HadoopUtil: Deleting 
hdfs://localhost:54310/user/hduser/nsl-fores
14/07/10 02:19:04 INFO mapreduce.BuildForest: Build Time: 0h 1m 20s 892
14/07/10 02:19:04 INFO mapreduce.BuildForest: Forest num Nodes: 4085
14/07/10 02:19:04 INFO mapreduce.BuildForest: Forest mean num Nodes: 408
14/07/10 02:19:04 INFO mapreduce.BuildForest: Forest mean max Depth: 12
14/07/10 02:19:04 INFO mapreduce.BuildForest: Storing the forest in: nsl-
fores/forest.seq
 
It gives the above statistics but not the oob error estimate as mentioned in 
the example here https://mahout.apache.org/users/classification/partial-
implementation.html
How can I get the oob?



RE: Mahout-0.7 BuildForest execution error

Posted by Stuti Awasthi <st...@hcl.com>.
Perfect.. It worked.. 

Thanks :)

-----Original Message-----
From: Lorenz Knies [mailto:lorenz.knies@metrigo.de] 
Sent: Monday, January 21, 2013 6:29 PM
To: user@mahout.apache.org
Subject: Re: Mahout-0.7 BuildForest execution error

i think the oob option is gone


On Jan 21, 2013, at 1:52 PM, Stuti Awasthi <st...@hcl.com> wrote:

> Hi,
> 
> I have downloaded Mahout and tried to execute Partial Implementation. When I try to run I am getting the parsing error:
> 
> $HADOOP_HOME/hadoop jar 
> $MAHOUT_HOME/examples/target/mahout-examples-0.7-job.jar 
> org.apache.mahout.classifier.df.mapreduce.BuildForest -oob -d 
> /testdata/KDDTrain+.arff -ds /testdata/KDDTrain+.info -sl 5 -p -t 100 
> -o /testdata/nsl-forest
> 
> 13/01/21 18:16:24 ERROR mapreduce.BuildForest: Exception
> org.apache.commons.cli2.OptionException: Unexpected /testdata/nsl-forest while processing Options
>        at org.apache.commons.cli2.commandline.Parser.parse(Parser.java:99)
>        at org.apache.mahout.classifier.df.mapreduce.BuildForest.run(BuildForest.java:139)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at org.apache.mahout.classifier.df.mapreduce.BuildForest.main(BuildForest.java:253)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>        at java.lang.reflect.Method.invoke(Method.java:616)
>        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Usage:
> [--data <path> --dataset <dataset> --selection <m> --no-complete 
> --minsplit <minsplit> --minprop <minprop> --seed <seed> --partial 
> --nbtrees <nbtrees> --output <path> --help] Options
>  --data (-d) path             Data path
>  --dataset (-ds) dataset      Dataset path
>  --selection (-sl) m          Optional, Number of variables to select randomly
>                               at each tree-node.
>                               For classification problem, the default is
>                               square root of the number of explanatory
>                               variables.
>                               For regression problem, the default is 1/3 of
>                               the number of explanatory variables.
>  --no-complete (-nc)          Optional, The tree is not complemented
>  --minsplit (-ms) minsplit    Optional, The tree-node is not divided, if the
>                               branching data size is smaller than this value.
>                               The default is 2.
>  --minprop (-mp) minprop      Optional, The tree-node is not divided, if the
>                               proportion of the variance of branching data is
>                               smaller than this value.
>                               In the case of a regression problem, this value
>                               is used. The default is 1/1000(0.001).
>  --seed (-sd) seed            Optional, seed value used to initialise the
>                               Random number generator
>  --partial (-p)               Optional, use the Partial Data implementation
>  --nbtrees (-t) nbtrees       Number of trees to grow
>  --output (-o) path           Output path, will contain the Decision Forest
>  --help (-h)                  Print out help
> 
> 
> If I try to run with Mahout-0.5, its working fine and generating /testdata/nsl-forest/forest.seq in hdfs.
> Is this a bug in Mahout-0.7 or am I doing something wrong.
> 
> Please Suggest
> 
> Thanks
> Stuti Awasthi
> 
> 
> 
> 
> 
> ::DISCLAIMER::
> ----------------------------------------------------------------------
> ----------------------------------------------------------------------
> --------
> 
> The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
> E-mail transmission is not guaranteed to be secure or error-free as 
> information could be intercepted, corrupted, lost, destroyed, arrive 
> late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
> Views or opinions, if any, presented in this email are solely those of 
> the author and may not necessarily reflect the views or opinions of 
> HCL or its affiliates. Any form of reproduction, dissemination, 
> copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
> Before opening any email and/or attachments, please check them for viruses and other defects.
> 
> ----------------------------------------------------------------------
> ----------------------------------------------------------------------
> --------


Re: Mahout-0.7 BuildForest execution error

Posted by Lorenz Knies <lo...@metrigo.de>.
i think the oob option is gone


On Jan 21, 2013, at 1:52 PM, Stuti Awasthi <st...@hcl.com> wrote:

> Hi,
> 
> I have downloaded Mahout and tried to execute Partial Implementation. When I try to run I am getting the parsing error:
> 
> $HADOOP_HOME/hadoop jar $MAHOUT_HOME/examples/target/mahout-examples-0.7-job.jar org.apache.mahout.classifier.df.mapreduce.BuildForest -oob -d /testdata/KDDTrain+.arff -ds /testdata/KDDTrain+.info -sl 5 -p -t 100 -o /testdata/nsl-forest
> 
> 13/01/21 18:16:24 ERROR mapreduce.BuildForest: Exception
> org.apache.commons.cli2.OptionException: Unexpected /testdata/nsl-forest while processing Options
>        at org.apache.commons.cli2.commandline.Parser.parse(Parser.java:99)
>        at org.apache.mahout.classifier.df.mapreduce.BuildForest.run(BuildForest.java:139)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at org.apache.mahout.classifier.df.mapreduce.BuildForest.main(BuildForest.java:253)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>        at java.lang.reflect.Method.invoke(Method.java:616)
>        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Usage:
> [--data <path> --dataset <dataset> --selection <m> --no-complete --minsplit
> <minsplit> --minprop <minprop> --seed <seed> --partial --nbtrees <nbtrees>
> --output <path> --help]
> Options
>  --data (-d) path             Data path
>  --dataset (-ds) dataset      Dataset path
>  --selection (-sl) m          Optional, Number of variables to select randomly
>                               at each tree-node.
>                               For classification problem, the default is
>                               square root of the number of explanatory
>                               variables.
>                               For regression problem, the default is 1/3 of
>                               the number of explanatory variables.
>  --no-complete (-nc)          Optional, The tree is not complemented
>  --minsplit (-ms) minsplit    Optional, The tree-node is not divided, if the
>                               branching data size is smaller than this value.
>                               The default is 2.
>  --minprop (-mp) minprop      Optional, The tree-node is not divided, if the
>                               proportion of the variance of branching data is
>                               smaller than this value.
>                               In the case of a regression problem, this value
>                               is used. The default is 1/1000(0.001).
>  --seed (-sd) seed            Optional, seed value used to initialise the
>                               Random number generator
>  --partial (-p)               Optional, use the Partial Data implementation
>  --nbtrees (-t) nbtrees       Number of trees to grow
>  --output (-o) path           Output path, will contain the Decision Forest
>  --help (-h)                  Print out help
> 
> 
> If I try to run with Mahout-0.5, its working fine and generating /testdata/nsl-forest/forest.seq in hdfs.
> Is this a bug in Mahout-0.7 or am I doing something wrong.
> 
> Please Suggest
> 
> Thanks
> Stuti Awasthi
> 
> 
> 
> 
> 
> ::DISCLAIMER::
> ----------------------------------------------------------------------------------------------------------------------------------------------------
> 
> The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
> E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
> (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
> Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
> views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
> distribution and / or publication of this message without the prior written consent of authorized representative of
> HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
> Before opening any email and/or attachments, please check them for viruses and other defects.
> 
> ----------------------------------------------------------------------------------------------------------------------------------------------------