You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Jeff Hansen <ds...@gmail.com> on 2011/08/18 00:27:03 UTC

Utility for dumping vectors to sparse csv triple files

Does anybody happen to know if there's already a utility out there for
dumping a sequence file of vectors to a csv file with vector,element,value?

I was hoping to shift some of my results over to R found a comment by Ted a
while back suggesting that the easiest method is to spit out sparse csv
triples and load them with

sparseMatrix(x=c(1,1,1,1), i=c(1,2,3,3), j=c(1,1,2,1))

from the Matrix library.

This wouldn't be that complicated to write, but I imagine I'm not the first
person to look for it.  If a utility like this doesn't already exist, does
anybody think it would be a worthwhile enhancement to add an option onto the
VectorDump utility to output to this format?  If so I'd be happy to offer up
a patch (although I might want to refactor the VectorHelper class to emit
straight out to the writer -- I'm not too fond of generating huge strings)

Re: Utility for dumping vectors to sparse csv triple files

Posted by Ted Dunning <te...@gmail.com>.
Actually, you can do almost exactly that.

There is a class called Dictionary that may help in converting row names to
indexes.  The intern() method takes a string and gives back the integer.
 Just read each row and use the inner loop in the sample code I posted to
convert the vectors that represent rows.

On Thu, Aug 18, 2011 at 7:53 AM, Jeff Hansen <ds...@gmail.com> wrote:

>  I suppose I can't just treat a file of named vectors
> as a matrix -- instead I imagine I need another utility to convert that
> vector file into a matrix file (with a dictionary file translating row
> numbers to vector names).
>

Re: Utility for dumping vectors to sparse csv triple files

Posted by Ted Dunning <te...@gmail.com>.
Yes.  Exactly.

JIRA allows files to be attached to JIRAs.  Just name your patch something
nice with the JIRA number in it.  Upload successive versions with the same
name so that JIRA knows that they are the same.

On Thu, Aug 18, 2011 at 7:53 AM, Jeff Hansen <ds...@gmail.com> wrote:

>  I just
> submitted my first jira the other day, what's the standard protocol for
> something like this? Submit the code as a svn patch file in a jira?
>

Re: Utility for dumping vectors to sparse csv triple files

Posted by Jeff Hansen <ds...@gmail.com>.
Yeah, I realized later in the game that I was actually thinking of sequence
files that contained text/vector pairs as matrices.  I haven't yet played
with the matrix manipulation utilities (well, not successfully), but that
gap may explain why.  I suppose I can't just treat a file of named vectors
as a matrix -- instead I imagine I need another utility to convert that
vector file into a matrix file (with a dictionary file translating row
numbers to vector names).

Right now I'm picturing the files that come out of the seq2sparse tool
intended for clustering purposes.  You get a dictionary that translates
columns to words, but you still have document names as rows.  I suppose in
the end it would be good to support two dictionary files -- one for rows and
one for columns (although if you ever transposed the matrix that could get
confusing, so maybe items/features would be a better generic name, or a
better yet a command line option for strongly naming row/column dict files).

So I suppose now I'm proposing two utilities -- one for converting vectors
into matrices (assuming one doesn't already exist) and another for spitting
out matrices in various formats (dense/sparse, horizontal rows/vertical
triples, specifiable delimiter/style).

One thing that would be really nice is a quick reference of standard input
and output file types for all the various utility functions.  A lot of them
are standard transforms and it would be nice to know what they're
transforming from and to at a glance (as well as what's available).

I'm getting pulled onto another project right now, but I have the house to
myself this weekend, so I should be able to work on something then -- I just
submitted my first jira the other day, what's the standard protocol for
something like this? Submit the code as a svn patch file in a jira?

On Wed, Aug 17, 2011 at 6:28 PM, Ted Dunning <te...@gmail.com> wrote:

> Sounds like a good idea in general.
>
> Here a tiny bit of code to get you rolling.  Adding this to the existing
> VectorDumper is better than using a standalone class as I have here.  Your
> thought about long strings is also very pertinent.
>
> public class DumpTriples {
>  public static void dump(PrintWriter out, Matrix m) {
>    for (MatrixSlice row : m) {
>      Iterator<Vector.Element> i = row.vector().iterateNonZero();
>      while (i.hasNext()) {
>        Vector.Element element = i.next();
>        out.printf("%d,%d,%f\n", row.index(), element.index(),
> element.get());
>       }
>    }
>  }
> }
>
>
> On Wed, Aug 17, 2011 at 3:27 PM, Jeff Hansen <ds...@gmail.com> wrote:
>
> > Does anybody happen to know if there's already a utility out there for
> > dumping a sequence file of vectors to a csv file with
> vector,element,value?
> >
> > I was hoping to shift some of my results over to R found a comment by Ted
> a
> > while back suggesting that the easiest method is to spit out sparse csv
> > triples and load them with
> >
> > sparseMatrix(x=c(1,1,1,1), i=c(1,2,3,3), j=c(1,1,2,1))
> >
> > from the Matrix library.
> >
> > This wouldn't be that complicated to write, but I imagine I'm not the
> first
> > person to look for it.  If a utility like this doesn't already exist,
> does
> > anybody think it would be a worthwhile enhancement to add an option onto
> > the
> > VectorDump utility to output to this format?  If so I'd be happy to offer
> > up
> > a patch (although I might want to refactor the VectorHelper class to emit
> > straight out to the writer -- I'm not too fond of generating huge
> strings)
> >
>

Re: Utility for dumping vectors to sparse csv triple files

Posted by Ted Dunning <te...@gmail.com>.
Sounds like a good idea in general.

Here a tiny bit of code to get you rolling.  Adding this to the existing
VectorDumper is better than using a standalone class as I have here.  Your
thought about long strings is also very pertinent.

public class DumpTriples {
  public static void dump(PrintWriter out, Matrix m) {
    for (MatrixSlice row : m) {
      Iterator<Vector.Element> i = row.vector().iterateNonZero();
      while (i.hasNext()) {
        Vector.Element element = i.next();
        out.printf("%d,%d,%f\n", row.index(), element.index(),
element.get());
      }
    }
  }
}


On Wed, Aug 17, 2011 at 3:27 PM, Jeff Hansen <ds...@gmail.com> wrote:

> Does anybody happen to know if there's already a utility out there for
> dumping a sequence file of vectors to a csv file with vector,element,value?
>
> I was hoping to shift some of my results over to R found a comment by Ted a
> while back suggesting that the easiest method is to spit out sparse csv
> triples and load them with
>
> sparseMatrix(x=c(1,1,1,1), i=c(1,2,3,3), j=c(1,1,2,1))
>
> from the Matrix library.
>
> This wouldn't be that complicated to write, but I imagine I'm not the first
> person to look for it.  If a utility like this doesn't already exist, does
> anybody think it would be a worthwhile enhancement to add an option onto
> the
> VectorDump utility to output to this format?  If so I'd be happy to offer
> up
> a patch (although I might want to refactor the VectorHelper class to emit
> straight out to the writer -- I'm not too fond of generating huge strings)
>