You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by adam35413 <ad...@gmail.com> on 2010/04/09 19:28:55 UTC
Dealing with kmean and meanshift output
I have been able to successfully run the kmean and meanshift examples on a
5-node Hadoop cluster. However, when it comes to dealing with the output, I
am a bit confused. I found the following page:
http://cwiki.apache.org/MAHOUT/viewing-results.html, but when I went to
track down the dictionary file I was unable to find it. Do I need to
generate the dictionary file separately or manually?
Thanks!
--
View this message in context: http://n3.nabble.com/Dealing-with-kmean-and-meanshift-output-tp708824p708824.html
Sent from the Mahout User List mailing list archive at Nabble.com.
Re: Dealing with kmean and meanshift output
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
adam35413 wrote:
> I did some testing and determined that a local copy of the results files AND
> an HDFS copy has to exist. From looking at the code, it appears that
> ClusterDump checks if the files exist locally at Line 129, but then tries to
> read the files from HDFS aroudn Line 135. If the files don't exist at both
> places mahout will bomb out.
>
> Whats the correct process to rebuild bin/mahout to make sure it is using the
> most recent version of the ClusterDump code?
>
I'd do a checkout from trunk and a clean install. I don't know why the
file handling you describe is working that way. Seems it might be a bug.
I would think the file locations should come with the path argument,
local or HDFS.
Re: Dealing with kmean and meanshift output
Posted by adam35413 <ad...@gmail.com>.
I did some testing and determined that a local copy of the results files AND
an HDFS copy has to exist. From looking at the code, it appears that
ClusterDump checks if the files exist locally at Line 129, but then tries to
read the files from HDFS aroudn Line 135. If the files don't exist at both
places mahout will bomb out.
Whats the correct process to rebuild bin/mahout to make sure it is using the
most recent version of the ClusterDump code?
--
View this message in context: http://n3.nabble.com/Dealing-with-kmean-and-meanshift-output-tp708824p713361.html
Sent from the Mahout User List mailing list archive at Nabble.com.
Re: Dealing with kmean and meanshift output
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
You might try running in a debugger, but it looks to me like there is
some problem with your sequenceFileDir file spec. The code is expecting
a directory containing the clusters, not a single file.
adam35413 wrote:
> I would agree; if $HADOOP_HOME is set, CONF should have been set as well. I
> fixed that environment variable, and now the error is:
>
> bin/mahout clusterdump --seqFileDir part-00000 --output testFile.txt
> running on hadoop, using
> HADOOP_HOME=/home/adam/Coding/hadoopResearch/hadoop-0.20.2 and
> HADOOP_CONF_DIR=/home/adam/Coding/hadoopResearch/hadoop-0.20.2/conf
> 10/04/09 15:11:08 ERROR driver.MahoutDriver: MahoutDriver failed with args:
> [--seqFileDir, part-00000, --output, testFile.txt, null]
> null
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.mahout.utils.clustering.ClusterDumper.printClusters(ClusterDumper.java:121)
> at
> org.apache.mahout.utils.clustering.ClusterDumper.main(ClusterDumper.java:302)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:172)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
>
Re: Dealing with kmean and meanshift output
Posted by adam35413 <ad...@gmail.com>.
I would agree; if $HADOOP_HOME is set, CONF should have been set as well. I
fixed that environment variable, and now the error is:
bin/mahout clusterdump --seqFileDir part-00000 --output testFile.txt
running on hadoop, using
HADOOP_HOME=/home/adam/Coding/hadoopResearch/hadoop-0.20.2 and
HADOOP_CONF_DIR=/home/adam/Coding/hadoopResearch/hadoop-0.20.2/conf
10/04/09 15:11:08 ERROR driver.MahoutDriver: MahoutDriver failed with args:
[--seqFileDir, part-00000, --output, testFile.txt, null]
null
Exception in thread "main" java.lang.NullPointerException
at
org.apache.mahout.utils.clustering.ClusterDumper.printClusters(ClusterDumper.java:121)
at
org.apache.mahout.utils.clustering.ClusterDumper.main(ClusterDumper.java:302)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:172)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
--
View this message in context: http://n3.nabble.com/Dealing-with-kmean-and-meanshift-output-tp708824p709066.html
Sent from the Mahout User List mailing list archive at Nabble.com.
Re: Dealing with kmean and meanshift output
Posted by Jake Mannix <ja...@gmail.com>.
We shouldn't need this, but can you try setting $HADOOP_CONF_DIR to
$HADOOP_HOME/conf and re-run? We should be automatically doing this
if HADOOP_CONF_DIR is not set.
-jake
On Fri, Apr 9, 2010 at 12:01 PM, adam35413 <ad...@gmail.com> wrote:
>
> I took a look at the code, and the only thing that seemed to be required
> was
> the Sequence file. I pulled the part-00000 file from output/clusterPoints/
> folder off of my Hadoop cluster, and tried the following command:
>
> bin/mahout clusterdump --seqFileDir part-00000 --output testFile.txt
>
> This resulted in the following error:
>
> no HADOOP_CONF_DIR or HADOOP_HOME set, running locally
> Apr 9, 2010 3:00:25 PM org.slf4j.impl.JCLLoggerAdapter error
> SEVERE: MahoutDriver failed with args: [--seqFileDir, part-00000, --output,
> testFile.txt, null]
> null
> Exception in thread "main" java.lang.NullPointerException
> at
>
> org.apache.mahout.utils.clustering.ClusterDumper.printClusters(ClusterDumper.java:121)
> at
>
> org.apache.mahout.utils.clustering.ClusterDumper.main(ClusterDumper.java:302)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
>
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> at
> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:172)
>
> Strange, since $HADOOP_HOME is actually set. Thoughts?
>
>
>
> Jeff Eastman wrote:
> >
> > The dictionary file contains a list (not sure how its delimited) of
> > element names for the input Vectors and is optional. See the new code in
> > trunk/utils in TestClusterDumper for some examples. I need to write test
> > sfor meanshift and also fuzzy kmeans to make sure they work but I
> > imagine they do. I also need to write tests that include the points, but
> > that appears to be done in memory so it likely won't scale to your
> > 5-node data set.
> >
> > Jeff
> >
> > adam35413 wrote:
> >> I have been able to successfully run the kmean and meanshift examples on
> >> a
> >> 5-node Hadoop cluster. However, when it comes to dealing with the
> >> output, I
> >> am a bit confused. I found the following page:
> >> http://cwiki.apache.org/MAHOUT/viewing-results.html, but when I went to
> >> track down the dictionary file I was unable to find it. Do I need to
> >> generate the dictionary file separately or manually?
> >>
> >> Thanks!
> >>
> >
> >
> >
> --
> View this message in context:
> http://n3.nabble.com/Dealing-with-kmean-and-meanshift-output-tp708824p709022.html
> Sent from the Mahout User List mailing list archive at Nabble.com.
>
Re: Dealing with kmean and meanshift output
Posted by adam35413 <ad...@gmail.com>.
I took a look at the code, and the only thing that seemed to be required was
the Sequence file. I pulled the part-00000 file from output/clusterPoints/
folder off of my Hadoop cluster, and tried the following command:
bin/mahout clusterdump --seqFileDir part-00000 --output testFile.txt
This resulted in the following error:
no HADOOP_CONF_DIR or HADOOP_HOME set, running locally
Apr 9, 2010 3:00:25 PM org.slf4j.impl.JCLLoggerAdapter error
SEVERE: MahoutDriver failed with args: [--seqFileDir, part-00000, --output,
testFile.txt, null]
null
Exception in thread "main" java.lang.NullPointerException
at
org.apache.mahout.utils.clustering.ClusterDumper.printClusters(ClusterDumper.java:121)
at
org.apache.mahout.utils.clustering.ClusterDumper.main(ClusterDumper.java:302)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:172)
Strange, since $HADOOP_HOME is actually set. Thoughts?
Jeff Eastman wrote:
>
> The dictionary file contains a list (not sure how its delimited) of
> element names for the input Vectors and is optional. See the new code in
> trunk/utils in TestClusterDumper for some examples. I need to write test
> sfor meanshift and also fuzzy kmeans to make sure they work but I
> imagine they do. I also need to write tests that include the points, but
> that appears to be done in memory so it likely won't scale to your
> 5-node data set.
>
> Jeff
>
> adam35413 wrote:
>> I have been able to successfully run the kmean and meanshift examples on
>> a
>> 5-node Hadoop cluster. However, when it comes to dealing with the
>> output, I
>> am a bit confused. I found the following page:
>> http://cwiki.apache.org/MAHOUT/viewing-results.html, but when I went to
>> track down the dictionary file I was unable to find it. Do I need to
>> generate the dictionary file separately or manually?
>>
>> Thanks!
>>
>
>
>
--
View this message in context: http://n3.nabble.com/Dealing-with-kmean-and-meanshift-output-tp708824p709022.html
Sent from the Mahout User List mailing list archive at Nabble.com.
Re: Dealing with kmean and meanshift output
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
The dictionary file contains a list (not sure how its delimited) of
element names for the input Vectors and is optional. See the new code in
trunk/utils in TestClusterDumper for some examples. I need to write test
sfor meanshift and also fuzzy kmeans to make sure they work but I
imagine they do. I also need to write tests that include the points, but
that appears to be done in memory so it likely won't scale to your
5-node data set.
Jeff
adam35413 wrote:
> I have been able to successfully run the kmean and meanshift examples on a
> 5-node Hadoop cluster. However, when it comes to dealing with the output, I
> am a bit confused. I found the following page:
> http://cwiki.apache.org/MAHOUT/viewing-results.html, but when I went to
> track down the dictionary file I was unable to find it. Do I need to
> generate the dictionary file separately or manually?
>
> Thanks!
>