You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Delroy Cameron <de...@gmail.com> on 2010/05/05 01:54:39 UTC
Dirichlet ClusterDump Output
so i've run Dirichlet Clustering using Mahout and i'm trying to see the
clusterdump. Of course i'm using a combination of ClusterDumper,
DirichletOutputState and DirichletCluster and TestL1ModelClustering to help
with the output.
so far i've successfully read each file in each state-x output folder. The
issue is that the vectors appear to be serialized as <Text,
DirichletCluster> pairs in each binary dump, which is fine. However, after
debugging it turns out that the model for each DirichletCluster is
null....and this make sense, since i'm reading from the dump file as
follows:
SequenceFile.Reader reader = new SequenceFile.Reader(fileSystem, inputPath,
conf);
Text key = (Text) reader.getKeyClass().newInstance();
DirichletCluster cluster = (DirichletCluster)
reader.getValueClass().newInstance();
i tried to set the fields for the DirichletCluster by using the following
method readFields(DataInput in);
DataInput istream = new DataInputStream(new FileInputStream(new
File(fileName)));
cluster.readFields(istream);
and i have a null pointer exception...
can i have a few suggestion on how to proceed here...
-----
--cheers
Delroy
--
View this message in context: http://lucene.472066.n3.nabble.com/Dirichlet-ClusterDump-Output-tp777637p777637.html
Sent from the Mahout User List mailing list archive at Nabble.com.
Re: Dirichlet ClusterDump Output
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
No, there is not currently a ClusterDumper test for LDA; I'm still
trying to figure out how to integrate it under Cluster. I'd love to see
your ideas on that!
Glad you are running again :)
On 5/5/10 2:55 AM, Delroy Cameron wrote:
> Jeff thanks a lot..you're right, i wasnt reading the clusters but merely
> instantiating DirichletCluster objects.
> with your suggestions the code works perfectly....i'll continue extending
> the code because i'm hoping to use LDA for clustering as well..i'm not sure
> that there is a TestClusterDumper for that. in any case, it may be a
> worthwhile exercise to keep getting my hands dirty a bit...
>
>
>
> -----
> --cheers
> Delroy
>
Re: Dirichlet ClusterDump Output
Posted by Delroy Cameron <de...@gmail.com>.
Jeff thanks a lot..you're right, i wasnt reading the clusters but merely
instantiating DirichletCluster objects.
with your suggestions the code works perfectly....i'll continue extending
the code because i'm hoping to use LDA for clustering as well..i'm not sure
that there is a TestClusterDumper for that. in any case, it may be a
worthwhile exercise to keep getting my hands dirty a bit...
-----
--cheers
Delroy
--
View this message in context: http://lucene.472066.n3.nabble.com/Dirichlet-ClusterDump-Output-tp777637p778372.html
Sent from the Mahout User List mailing list archive at Nabble.com.
Re: Dirichlet ClusterDump Output
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
Hi Delroy,
You did not say if you were using 0.3 or trunk; I suggest trunk since it
has been recently better integrated with Dirichlet. Looking at your code
fragment and comparing it with what the ClusterDumper is (now) doing:
SequenceFile.Reader reader = new SequenceFile.Reader(fs, path, conf);
Writable key = (Writable) reader.getKeyClass().newInstance();
Writable value = (Writable) reader.getValueClass().newInstance();
while (reader.next(key, value)) {
Cluster cluster = (Cluster) value;
String fmtStr = useJSON ? cluster.asJsonString() :
cluster.asFormatString(dictionary);
... it kinda looks like you are not actually reading in the clusters and
their models; rather just creating a new instance of DirichletCluster
(the value class). This approach will not read in the model or any of
the cluster state, hence your observations. You should be able to just
run the ClusterDumper by pointing at your cluster directory as in
TestClusterDumper.testDirichlet.
If you really want to write your own code for reading the clusters, I
suggest copying the above and remembering to create a new value object
in your loop otherwise the first instance will be reused by the reader
and you will end up with all your clusters being identical. Something
like this:
while (reader.next(key, value)) {
DirichletCluster cluster = (DirichletCluster) value;
String fmtStr = useJSON ? cluster.asJsonString() :
cluster.asFormatString(dictionary);
<save the cluster in some data structure>
value = (Writable) reader.getValueClass().newInstance();
}
Let me know how it goes,
Jeff
On 5/4/10 5:54 PM, Delroy Cameron wrote:
> so i've run Dirichlet Clustering using Mahout and i'm trying to see the
> clusterdump. Of course i'm using a combination of ClusterDumper,
> DirichletOutputState and DirichletCluster and TestL1ModelClustering to help
> with the output.
>
> so far i've successfully read each file in each state-x output folder. The
> issue is that the vectors appear to be serialized as<Text,
> DirichletCluster> pairs in each binary dump, which is fine. However, after
> debugging it turns out that the model for each DirichletCluster is
> null....and this make sense, since i'm reading from the dump file as
> follows:
>
> SequenceFile.Reader reader = new SequenceFile.Reader(fileSystem, inputPath,
> conf);
> Text key = (Text) reader.getKeyClass().newInstance();
> DirichletCluster cluster = (DirichletCluster)
> reader.getValueClass().newInstance();
>
> i tried to set the fields for the DirichletCluster by using the following
> method readFields(DataInput in);
> DataInput istream = new DataInputStream(new FileInputStream(new
> File(fileName)));
> cluster.readFields(istream);
>
> and i have a null pointer exception...
>
> can i have a few suggestion on how to proceed here...
>
> -----
> --cheers
> Delroy
>