You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Alan Krumholz <al...@yahoo.com.mx> on 2013/09/05 03:58:44 UTC
using KmeansDriver with HDFS
Hi all,
I'm using mahout 0.7
I'm trying to use KmeansDriver (org.apache.mahout.clustering.kmeans.KMeansDriver) with HDFS and I'm having some issues.
When I use it with my local file system everything seems to be working fine.
However, as soon as I change the Configuration object to use HDFS:
Configuration conf = new Configuration();
conf.addResource(new Path("C:\\hdp-win\\hadoop\\hadoop-1.1.0-SNAPSHOT\\conf\\core-site.xml"));
conf.addResource(new Path("C:\\hdp-win\\hadoop\\hadoop-1.1.0-SNAPSHOT\\conf\\hdfs-site.xml"))
I run into problems
I was looking at the exception I get:
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.RangeCheck(ArrayList.java:547)
at java.util.ArrayList.get(ArrayList.java:322)
at org.apache.mahout.clustering.classify.ClusterClassifier.readFromSeqFiles(ClusterClassifier.java:215)
I pulled that code (org.apache.mahout.clustering.classify.ClusterClassifier.readFromSeqFiles(ClusterClassifier.java:215)and I think is trying to read a file from one of the paths I passed to the method but with a new instance of the configuration object (not the configuration object I passed to the method but one that doesn't have my HDFS configured)
205
public void [More ...] readFromSeqFiles(Configuration conf, Path path) throws IOException {
206
Configuration config = new Configuration();
207
List<Cluster> clusters = Lists.newArrayList();
208
for (ClusterWritable cw : new SequenceFileDirValueIterable<ClusterWritable>(path, PathType.LIST,
209
PathFilters.logsCRCFilter(), config)) {
210
Cluster cluster = cw.getValue();
211
cluster.configure(conf);
212
clusters.add(cluster);
213
}
214
this.models = clusters;
215
modelClass = models.get(0).getClass().getName();
216
this.policy = readPolicy(path);
217
}
any help would be really appreciated :)
Thanks!
Alan
Re: using KmeansDriver with HDFS
Posted by Taner Diler <ta...@gmail.com>.
With 0.8 you can set conf files as
Configuration conf = new Configuration();
String HADOOP_HOME = System.getenv("HADOOP_PREFIX");
conf.addResource(new Path(HADOOP_HOME, "conf/core-site.xml"));
conf.addResource(new Path(HADOOP_HOME, "conf/hdfs-site.xml"));
conf.addResource(new Path(HADOOP_HOME, "conf/mapred-site.xml"));
but you must define HADOOP_PREFIX as hadoop path.
On Thu, Sep 5, 2013 at 9:07 AM, Ted Dunning <te...@gmail.com> wrote:
> On Wed, Sep 4, 2013 at 6:58 PM, Alan Krumholz <alan_krumholz@yahoo.com.mx
> >wrote:
>
> > I pulled that code
> >
> (org.apache.mahout.clustering.classify.ClusterClassifier.readFromSeqFiles(ClusterClassifier.java:215)and
> > I think is trying to read a file from one of the paths I passed to the
> > method but with a new instance of the configuration object (not the
> > configuration object I passed to the method but one that doesn't have my
> > HDFS configured)
> >
>
>
> This is quite plausibly a bug. This is a common error when using the HDFS
> API.
>
> Have you checked what happens with 0.8?
>
Re: using KmeansDriver with HDFS
Posted by Suneel Marthi <su...@yahoo.com>.
This is definitely a bug (and seems to have been there for a while now) . Please do file a JIRA.
________________________________
From: Alan Krumholz <al...@yahoo.com.mx>
To: "user@mahout.apache.org" <us...@mahout.apache.org>
Sent: Thursday, September 5, 2013 2:00 PM
Subject: Re: using KmeansDriver with HDFS
I just replaced the jars in my project with mahout 0.8 jars and I get the same exception:
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.RangeCheck(ArrayList.java:547)
at java.util.ArrayList.get(ArrayList.java:322)
at org.apache.mahout.clustering.classify.ClusterClassifier.readFromSeqFiles(ClusterClassifier.java:215)
should I file a bug?
Thanks!
Alan
________________________________
De: Ted Dunning <te...@gmail.com>
Para: "user@mahout.apache.org" <us...@mahout.apache.org>; Alan Krumholz <al...@yahoo.com.mx>
Enviado: Miércoles, 4 de septiembre, 2013 23:07:02
Asunto: Re: using KmeansDriver with HDFS
On Wed, Sep 4, 2013 at 6:58 PM, Alan Krumholz <al...@yahoo.com.mx>wrote:
> I pulled that code
> (org.apache.mahout.clustering.classify.ClusterClassifier.readFromSeqFiles(ClusterClassifier.java:215)and
> I think is trying to read a file from one of the paths I passed to the
> method but with a new instance of the configuration object (not the
> configuration object I passed to the method but one that doesn't have my
> HDFS configured)
>
This is quite plausibly a bug. This is a common error when using the HDFS
API.
Have you checked what happens with 0.8?
Re: using KmeansDriver with HDFS
Posted by Alan Krumholz <al...@yahoo.com.mx>.
I just replaced the jars in my project with mahout 0.8 jars and I get the same exception:
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.RangeCheck(ArrayList.java:547)
at java.util.ArrayList.get(ArrayList.java:322)
at org.apache.mahout.clustering.classify.ClusterClassifier.readFromSeqFiles(ClusterClassifier.java:215)
should I file a bug?
Thanks!
Alan
________________________________
De: Ted Dunning <te...@gmail.com>
Para: "user@mahout.apache.org" <us...@mahout.apache.org>; Alan Krumholz <al...@yahoo.com.mx>
Enviado: Miércoles, 4 de septiembre, 2013 23:07:02
Asunto: Re: using KmeansDriver with HDFS
On Wed, Sep 4, 2013 at 6:58 PM, Alan Krumholz <al...@yahoo.com.mx>wrote:
> I pulled that code
> (org.apache.mahout.clustering.classify.ClusterClassifier.readFromSeqFiles(ClusterClassifier.java:215)and
> I think is trying to read a file from one of the paths I passed to the
> method but with a new instance of the configuration object (not the
> configuration object I passed to the method but one that doesn't have my
> HDFS configured)
>
This is quite plausibly a bug. This is a common error when using the HDFS
API.
Have you checked what happens with 0.8?
Re: using KmeansDriver with HDFS
Posted by Ted Dunning <te...@gmail.com>.
On Wed, Sep 4, 2013 at 6:58 PM, Alan Krumholz <al...@yahoo.com.mx>wrote:
> I pulled that code
> (org.apache.mahout.clustering.classify.ClusterClassifier.readFromSeqFiles(ClusterClassifier.java:215)and
> I think is trying to read a file from one of the paths I passed to the
> method but with a new instance of the configuration object (not the
> configuration object I passed to the method but one that doesn't have my
> HDFS configured)
>
This is quite plausibly a bug. This is a common error when using the HDFS
API.
Have you checked what happens with 0.8?