You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Stefan Wienert <st...@wienert.cc> on 2011/05/25 13:59:41 UTC
How to convert SequenceFile to SequenceFile?
Hi,
I need some help using Hadoop :
I'm trying to do some Dimensional reduction after this tutorial:
https://cwiki.apache.org/confluence/display/MAHOUT/Dimensional+Reduction
I created my tf-idf-vectors from text saved in lucene:
https://cwiki.apache.org/confluence/display/MAHOUT/Creating+Vectors+from+Text
Now, I got a problem when I try to transpose the tf-idf-vectors:
java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot
be cast to org.apache.hadoop.io.IntWritable
So, my question:
What is the easiest way to get my data from
SequenceFile<LongWritable,VectorWritable> to
SequenceFile<IntWritable,VectorWritable>?
Or can I create this directly from a Lucene-Index?
Thx
Stefan
Re: How to convert SequenceFile to SequenceFile?
Posted by Ted Dunning <te...@gmail.com>.
Ahh... sorry. That was my first impression, but then somebody said values
(meaning *KEY* values, of course) and I jumped the tracks.
On Wed, May 25, 2011 at 2:58 PM, Jake Mannix <ja...@gmail.com> wrote:
> We're dealing with the keys, not the values. But yes, the binary case, you
> don't
> even need matrix entries. Just the keys.
>
> On Wed, May 25, 2011 at 2:49 PM, Ted Dunning <te...@gmail.com>
> wrote:
>
> > The limiting case here is binary matrices where it really would be nice
> to
> > have vectors be 20-30x smaller. These are a common case.
> >
> > On Wed, May 25, 2011 at 2:28 PM, Jake Mannix <ja...@gmail.com>
> > wrote:
> >
> > > On Wed, May 25, 2011 at 2:23 PM, Sean Owen <sr...@gmail.com> wrote:
> > >
> > > > Variable-length saves space for values under about 2^21 ~= 2M. It's a
> > > wash
> > > > for values up to about 2^28 ~= 268M. It costs an extra byte for
> larger
> > > > values. I'm thinking unsigned values here at the moment, and ignoring
> > the
> > > > CPU costs of encoding/decoding, which is tiny.
> > > >
> > > > Yes it's a loss for 15/16ths of the key space. My big assumption is
> > that
> > > in
> > > > many cases that first 1/16th is heavily used. It's certainly true
> when
> > > > values are counts, and true when they're product IDs. When they're
> > > hashes,
> > > > nope.
> > > >
> > >
> > > Ok, good point: when they're counts, they're probably a huge savings.
> > > Those
> > > are often very small.
> > >
> > > For IDs, the DistributedRowMatrix probably shouldn't even be used for
> > > numbers
> > > of rows under the 10's of millions - otherwise you can probably fit in
> > > memory.
> > >
> > > -jake
> > >
> >
>
Re: How to convert SequenceFile to SequenceFile?
Posted by Jake Mannix <ja...@gmail.com>.
We're dealing with the keys, not the values. But yes, the binary case, you
don't
even need matrix entries. Just the keys.
On Wed, May 25, 2011 at 2:49 PM, Ted Dunning <te...@gmail.com> wrote:
> The limiting case here is binary matrices where it really would be nice to
> have vectors be 20-30x smaller. These are a common case.
>
> On Wed, May 25, 2011 at 2:28 PM, Jake Mannix <ja...@gmail.com>
> wrote:
>
> > On Wed, May 25, 2011 at 2:23 PM, Sean Owen <sr...@gmail.com> wrote:
> >
> > > Variable-length saves space for values under about 2^21 ~= 2M. It's a
> > wash
> > > for values up to about 2^28 ~= 268M. It costs an extra byte for larger
> > > values. I'm thinking unsigned values here at the moment, and ignoring
> the
> > > CPU costs of encoding/decoding, which is tiny.
> > >
> > > Yes it's a loss for 15/16ths of the key space. My big assumption is
> that
> > in
> > > many cases that first 1/16th is heavily used. It's certainly true when
> > > values are counts, and true when they're product IDs. When they're
> > hashes,
> > > nope.
> > >
> >
> > Ok, good point: when they're counts, they're probably a huge savings.
> > Those
> > are often very small.
> >
> > For IDs, the DistributedRowMatrix probably shouldn't even be used for
> > numbers
> > of rows under the 10's of millions - otherwise you can probably fit in
> > memory.
> >
> > -jake
> >
>
Re: How to convert SequenceFile to SequenceFile?
Posted by Ted Dunning <te...@gmail.com>.
The limiting case here is binary matrices where it really would be nice to
have vectors be 20-30x smaller. These are a common case.
On Wed, May 25, 2011 at 2:28 PM, Jake Mannix <ja...@gmail.com> wrote:
> On Wed, May 25, 2011 at 2:23 PM, Sean Owen <sr...@gmail.com> wrote:
>
> > Variable-length saves space for values under about 2^21 ~= 2M. It's a
> wash
> > for values up to about 2^28 ~= 268M. It costs an extra byte for larger
> > values. I'm thinking unsigned values here at the moment, and ignoring the
> > CPU costs of encoding/decoding, which is tiny.
> >
> > Yes it's a loss for 15/16ths of the key space. My big assumption is that
> in
> > many cases that first 1/16th is heavily used. It's certainly true when
> > values are counts, and true when they're product IDs. When they're
> hashes,
> > nope.
> >
>
> Ok, good point: when they're counts, they're probably a huge savings.
> Those
> are often very small.
>
> For IDs, the DistributedRowMatrix probably shouldn't even be used for
> numbers
> of rows under the 10's of millions - otherwise you can probably fit in
> memory.
>
> -jake
>
Re: How to convert SequenceFile to SequenceFile?
Posted by Jake Mannix <ja...@gmail.com>.
On Wed, May 25, 2011 at 2:23 PM, Sean Owen <sr...@gmail.com> wrote:
> Variable-length saves space for values under about 2^21 ~= 2M. It's a wash
> for values up to about 2^28 ~= 268M. It costs an extra byte for larger
> values. I'm thinking unsigned values here at the moment, and ignoring the
> CPU costs of encoding/decoding, which is tiny.
>
> Yes it's a loss for 15/16ths of the key space. My big assumption is that in
> many cases that first 1/16th is heavily used. It's certainly true when
> values are counts, and true when they're product IDs. When they're hashes,
> nope.
>
Ok, good point: when they're counts, they're probably a huge savings. Those
are often very small.
For IDs, the DistributedRowMatrix probably shouldn't even be used for
numbers
of rows under the 10's of millions - otherwise you can probably fit in
memory.
-jake
Re: How to convert SequenceFile to SequenceFile?
Posted by Sean Owen <sr...@gmail.com>.
Variable-length saves space for values under about 2^21 ~= 2M. It's a wash
for values up to about 2^28 ~= 268M. It costs an extra byte for larger
values. I'm thinking unsigned values here at the moment, and ignoring the
CPU costs of encoding/decoding, which is tiny.
Yes it's a loss for 15/16ths of the key space. My big assumption is that in
many cases that first 1/16th is heavily used. It's certainly true when
values are counts, and true when they're product IDs. When they're hashes,
nope.
I actually don't know the situation here -- if it's a bad idea we don't do
it. But there are surely places where it's a good idea.
On Wed, May 25, 2011 at 10:14 PM, Jake Mannix <ja...@gmail.com> wrote:
>
> If you have more than a 32M IDs *total*, even if they are sequential
> starting
> at 0 "most" of them will take up the full 4 bytes, and only a tiny fraction
> will
> take up less than 3.
>
Re: How to convert SequenceFile to SequenceFile?
Posted by Jake Mannix <ja...@gmail.com>.
On Wed, May 25, 2011 at 1:52 PM, Sean Owen <sr...@gmail.com> wrote:
> If keys are distributed across the keyspace then yes it is a net loss to
> try
> variable-length encoding. However it's my impression that keys aren't in
> many contexts. (I actually haven't thought about this one hard.)
>
> But for example in recommender-land where keys are product IDs, it's more
> common for there to be millions of keys ranging in value to, well, a few
> million, than spread across the key space.
>
If you have more than a 32M IDs *total*, even if they are sequential
starting
at 0 "most" of them will take up the full 4 bytes, and only a tiny fraction
will
take up less than 3.
Re: How to convert SequenceFile to SequenceFile?
Posted by Sean Owen <sr...@gmail.com>.
If keys are distributed across the keyspace then yes it is a net loss to try
variable-length encoding. However it's my impression that keys aren't in
many contexts. (I actually haven't thought about this one hard.)
But for example in recommender-land where keys are product IDs, it's more
common for there to be millions of keys ranging in value to, well, a few
million, than spread across the key space.
On Wed, May 25, 2011 at 9:37 PM, Jake Mannix <ja...@gmail.com> wrote:
> On Wed, May 25, 2011 at 1:33 PM, Sean Owen <sr...@gmail.com> wrote:
>
> > (I suggest we not use IntWritable or LongWritable, but favor
> VarIntWritable
> > and VarLongWritable, which are variable length encoding versions, where
> > possible. Saving a couple bytes per key adds up.)
> >
>
> If you have millions to hundreds of millions of keys, how many of them are
> going to be low enough to fit in less than 4 bytes? As soon as you have
> more than 16 million, "most" numbers take up the full 4 bytes, right?
>
> -jake
>
Re: How to convert SequenceFile to SequenceFile?
Posted by Jake Mannix <ja...@gmail.com>.
On Wed, May 25, 2011 at 1:33 PM, Sean Owen <sr...@gmail.com> wrote:
> (I suggest we not use IntWritable or LongWritable, but favor VarIntWritable
> and VarLongWritable, which are variable length encoding versions, where
> possible. Saving a couple bytes per key adds up.)
>
If you have millions to hundreds of millions of keys, how many of them are
going to be low enough to fit in less than 4 bytes? As soon as you have
more than 16 million, "most" numbers take up the full 4 bytes, right?
-jake
Re: How to convert SequenceFile to SequenceFile?
Posted by Sean Owen <sr...@gmail.com>.
(I suggest we not use IntWritable or LongWritable, but favor VarIntWritable
and VarLongWritable, which are variable length encoding versions, where
possible. Saving a couple bytes per key adds up.)
On Wed, May 25, 2011 at 5:38 PM, Jake Mannix <ja...@gmail.com> wrote:
> Hmm... then it looks like we're spitting out Long ids from the
> lucene.vector
> output.
>
> Take a look at RowIdJob - it takes SequenceFile<Text,VectorWritable> and
> converts
> it to a pair of sequence files: SequenceFile<IntWritable, VectorWritable>,
> and
> SequenceFile<IntWritable,Text> (the latter being the "dictionary" of what
> int ids
> correspond to what original text ids). This job could be modified
> trivially
> by swapping
> every reference to Text to LongWritable.
>
> On Wed, May 25, 2011 at 8:00 AM, Stefan Wienert <st...@wienert.cc> wrote:
>
> > Yes, with:
> > bin/mahout lucene.vector \
> > --dir /home/hadoop/MahoutStatements/tf_index \
> > --field fulltext \
> > --dictOut /home/hadoop/MahoutStatements/dict.txt \
> > --output /home/hadoop/MahoutStatements/tfidf-vectors \
> > --idField id \
> > --weight TFIDF
> >
> > 2011/5/25 Jake Mannix <ja...@gmail.com>:
> > > Did you rebuild your tfidf-vectors with trunk as well?
> > >
> > > On Wed, May 25, 2011 at 6:59 AM, Stefan Wienert <st...@wienert.cc>
> > wrote:
> > >
> > >> First, I use http://svn.apache.org/repos/asf/mahout/trunk, tested
> some
> > >> minutes ago with the newest version.
> > >>
> > >> And still:
> > >> bin/mahout transpose \
> > >> --input /home/hadoop/MahoutStatements/tfidf-vectors \
> > >> --numRows 227 \
> > >> --numCols 107909 \
> > >> --tempDir /home/hadoop/MahoutStatements/tfidf-matrix/transpose
> > >> produces:
> > >> java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot
> > >> be cast to org.apache.hadoop.io.IntWritable
> > >>
> > >> My first idea to change "lucene.vector" does not work, there is too
> > >> much to change.
> > >>
> > >> So... Ideas? What about changing "transpose" and "matrixmult" to use
> > >> LongWritable instead of IntWritable? Is this problematically?
> > >>
> > >> 2011/5/25 Jake Mannix <ja...@gmail.com>:
> > >> > On Wed, May 25, 2011 at 6:14 AM, Stefan Wienert <st...@wienert.cc>
> > >> wrote:
> > >> >
> > >> >> So the real problem is, that "transpose" and "matrixmult" (maybe)
> > >> >> still uses IntWritable instead of LongWritable".
> > >> >>
> > >> >
> > >> > It's the other way around: matrix operations use keys which are
> ints,
> > and
> > >> > the lucene.vector class needs to respect this. It doesn't on
> current
> > >> trunk?
> > >> >
> > >> > -jake
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Stefan Wienert
> > >>
> > >> http://www.wienert.cc
> > >> stefan@wienert.cc
> > >>
> > >> Telefon: +495251-2026838 (neue Nummer seit 20.06.10)
> > >> Mobil: +49176-40170270
> > >>
> > >
> >
> >
> >
> > --
> > Stefan Wienert
> >
> > http://www.wienert.cc
> > stefan@wienert.cc
> >
> > Telefon: +495251-2026838 (neue Nummer seit 20.06.10)
> > Mobil: +49176-40170270
> >
>
Re: How to convert SequenceFile to SequenceFile?
Posted by Jake Mannix <ja...@gmail.com>.
Hmm... then it looks like we're spitting out Long ids from the lucene.vector
output.
Take a look at RowIdJob - it takes SequenceFile<Text,VectorWritable> and
converts
it to a pair of sequence files: SequenceFile<IntWritable, VectorWritable>,
and
SequenceFile<IntWritable,Text> (the latter being the "dictionary" of what
int ids
correspond to what original text ids). This job could be modified trivially
by swapping
every reference to Text to LongWritable.
On Wed, May 25, 2011 at 8:00 AM, Stefan Wienert <st...@wienert.cc> wrote:
> Yes, with:
> bin/mahout lucene.vector \
> --dir /home/hadoop/MahoutStatements/tf_index \
> --field fulltext \
> --dictOut /home/hadoop/MahoutStatements/dict.txt \
> --output /home/hadoop/MahoutStatements/tfidf-vectors \
> --idField id \
> --weight TFIDF
>
> 2011/5/25 Jake Mannix <ja...@gmail.com>:
> > Did you rebuild your tfidf-vectors with trunk as well?
> >
> > On Wed, May 25, 2011 at 6:59 AM, Stefan Wienert <st...@wienert.cc>
> wrote:
> >
> >> First, I use http://svn.apache.org/repos/asf/mahout/trunk, tested some
> >> minutes ago with the newest version.
> >>
> >> And still:
> >> bin/mahout transpose \
> >> --input /home/hadoop/MahoutStatements/tfidf-vectors \
> >> --numRows 227 \
> >> --numCols 107909 \
> >> --tempDir /home/hadoop/MahoutStatements/tfidf-matrix/transpose
> >> produces:
> >> java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot
> >> be cast to org.apache.hadoop.io.IntWritable
> >>
> >> My first idea to change "lucene.vector" does not work, there is too
> >> much to change.
> >>
> >> So... Ideas? What about changing "transpose" and "matrixmult" to use
> >> LongWritable instead of IntWritable? Is this problematically?
> >>
> >> 2011/5/25 Jake Mannix <ja...@gmail.com>:
> >> > On Wed, May 25, 2011 at 6:14 AM, Stefan Wienert <st...@wienert.cc>
> >> wrote:
> >> >
> >> >> So the real problem is, that "transpose" and "matrixmult" (maybe)
> >> >> still uses IntWritable instead of LongWritable".
> >> >>
> >> >
> >> > It's the other way around: matrix operations use keys which are ints,
> and
> >> > the lucene.vector class needs to respect this. It doesn't on current
> >> trunk?
> >> >
> >> > -jake
> >> >
> >>
> >>
> >>
> >> --
> >> Stefan Wienert
> >>
> >> http://www.wienert.cc
> >> stefan@wienert.cc
> >>
> >> Telefon: +495251-2026838 (neue Nummer seit 20.06.10)
> >> Mobil: +49176-40170270
> >>
> >
>
>
>
> --
> Stefan Wienert
>
> http://www.wienert.cc
> stefan@wienert.cc
>
> Telefon: +495251-2026838 (neue Nummer seit 20.06.10)
> Mobil: +49176-40170270
>
Re: How to convert SequenceFile to SequenceFile?
Posted by Stefan Wienert <st...@wienert.cc>.
Yes, with:
bin/mahout lucene.vector \
--dir /home/hadoop/MahoutStatements/tf_index \
--field fulltext \
--dictOut /home/hadoop/MahoutStatements/dict.txt \
--output /home/hadoop/MahoutStatements/tfidf-vectors \
--idField id \
--weight TFIDF
2011/5/25 Jake Mannix <ja...@gmail.com>:
> Did you rebuild your tfidf-vectors with trunk as well?
>
> On Wed, May 25, 2011 at 6:59 AM, Stefan Wienert <st...@wienert.cc> wrote:
>
>> First, I use http://svn.apache.org/repos/asf/mahout/trunk, tested some
>> minutes ago with the newest version.
>>
>> And still:
>> bin/mahout transpose \
>> --input /home/hadoop/MahoutStatements/tfidf-vectors \
>> --numRows 227 \
>> --numCols 107909 \
>> --tempDir /home/hadoop/MahoutStatements/tfidf-matrix/transpose
>> produces:
>> java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot
>> be cast to org.apache.hadoop.io.IntWritable
>>
>> My first idea to change "lucene.vector" does not work, there is too
>> much to change.
>>
>> So... Ideas? What about changing "transpose" and "matrixmult" to use
>> LongWritable instead of IntWritable? Is this problematically?
>>
>> 2011/5/25 Jake Mannix <ja...@gmail.com>:
>> > On Wed, May 25, 2011 at 6:14 AM, Stefan Wienert <st...@wienert.cc>
>> wrote:
>> >
>> >> So the real problem is, that "transpose" and "matrixmult" (maybe)
>> >> still uses IntWritable instead of LongWritable".
>> >>
>> >
>> > It's the other way around: matrix operations use keys which are ints, and
>> > the lucene.vector class needs to respect this. It doesn't on current
>> trunk?
>> >
>> > -jake
>> >
>>
>>
>>
>> --
>> Stefan Wienert
>>
>> http://www.wienert.cc
>> stefan@wienert.cc
>>
>> Telefon: +495251-2026838 (neue Nummer seit 20.06.10)
>> Mobil: +49176-40170270
>>
>
--
Stefan Wienert
http://www.wienert.cc
stefan@wienert.cc
Telefon: +495251-2026838 (neue Nummer seit 20.06.10)
Mobil: +49176-40170270
Re: How to convert SequenceFile to SequenceFile?
Posted by Jake Mannix <ja...@gmail.com>.
Did you rebuild your tfidf-vectors with trunk as well?
On Wed, May 25, 2011 at 6:59 AM, Stefan Wienert <st...@wienert.cc> wrote:
> First, I use http://svn.apache.org/repos/asf/mahout/trunk, tested some
> minutes ago with the newest version.
>
> And still:
> bin/mahout transpose \
> --input /home/hadoop/MahoutStatements/tfidf-vectors \
> --numRows 227 \
> --numCols 107909 \
> --tempDir /home/hadoop/MahoutStatements/tfidf-matrix/transpose
> produces:
> java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot
> be cast to org.apache.hadoop.io.IntWritable
>
> My first idea to change "lucene.vector" does not work, there is too
> much to change.
>
> So... Ideas? What about changing "transpose" and "matrixmult" to use
> LongWritable instead of IntWritable? Is this problematically?
>
> 2011/5/25 Jake Mannix <ja...@gmail.com>:
> > On Wed, May 25, 2011 at 6:14 AM, Stefan Wienert <st...@wienert.cc>
> wrote:
> >
> >> So the real problem is, that "transpose" and "matrixmult" (maybe)
> >> still uses IntWritable instead of LongWritable".
> >>
> >
> > It's the other way around: matrix operations use keys which are ints, and
> > the lucene.vector class needs to respect this. It doesn't on current
> trunk?
> >
> > -jake
> >
>
>
>
> --
> Stefan Wienert
>
> http://www.wienert.cc
> stefan@wienert.cc
>
> Telefon: +495251-2026838 (neue Nummer seit 20.06.10)
> Mobil: +49176-40170270
>
Re: How to convert SequenceFile to SequenceFile?
Posted by Stefan Wienert <st...@wienert.cc>.
First, I use http://svn.apache.org/repos/asf/mahout/trunk, tested some
minutes ago with the newest version.
And still:
bin/mahout transpose \
--input /home/hadoop/MahoutStatements/tfidf-vectors \
--numRows 227 \
--numCols 107909 \
--tempDir /home/hadoop/MahoutStatements/tfidf-matrix/transpose
produces:
java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot
be cast to org.apache.hadoop.io.IntWritable
My first idea to change "lucene.vector" does not work, there is too
much to change.
So... Ideas? What about changing "transpose" and "matrixmult" to use
LongWritable instead of IntWritable? Is this problematically?
2011/5/25 Jake Mannix <ja...@gmail.com>:
> On Wed, May 25, 2011 at 6:14 AM, Stefan Wienert <st...@wienert.cc> wrote:
>
>> So the real problem is, that "transpose" and "matrixmult" (maybe)
>> still uses IntWritable instead of LongWritable".
>>
>
> It's the other way around: matrix operations use keys which are ints, and
> the lucene.vector class needs to respect this. It doesn't on current trunk?
>
> -jake
>
--
Stefan Wienert
http://www.wienert.cc
stefan@wienert.cc
Telefon: +495251-2026838 (neue Nummer seit 20.06.10)
Mobil: +49176-40170270
Re: How to convert SequenceFile to SequenceFile?
Posted by Jake Mannix <ja...@gmail.com>.
On Wed, May 25, 2011 at 6:14 AM, Stefan Wienert <st...@wienert.cc> wrote:
> So the real problem is, that "transpose" and "matrixmult" (maybe)
> still uses IntWritable instead of LongWritable".
>
It's the other way around: matrix operations use keys which are ints, and
the lucene.vector class needs to respect this. It doesn't on current trunk?
-jake
Re: How to convert SequenceFile to SequenceFile?
Posted by Stefan Wienert <st...@wienert.cc>.
So the real problem is, that "transpose" and "matrixmult" (maybe)
still uses IntWritable instead of LongWritable".
Still, int should be enough, the easiest approach for that is to
modify "lucene.vector" so that it creates
SequenceFile<IntWritable,VectorWritable>
Or is there a "transpose" and "matrixmult" for <LongWritable,VectorWritable>?
2011/5/25 Sean Owen <sr...@gmail.com>:
> (This wiki looks well out of date, recent as of only version 0.2. For
> example we don't use IntWritable anymore. It needs an update, but,
> ought not be too different as of now, version 0.5.)
>
> You would need to key by int instead of long, conceptually. Do you
> really have 32-bit values? then just write out an int in a Writable,
> not long.
>
> If not, you can just hash. I use elsewhere what Java uses: value ^
> (value >>> 32). Actually I also add "& 0x7FFFFFFF" to make sure it's
> positive.
>
> You would need to store the reverse mapping from int to long to get
> your original values out later. And there is a tiny chance of
> collision.
>
> On Wed, May 25, 2011 at 12:59 PM, Stefan Wienert <st...@wienert.cc> wrote:
>> Hi,
>>
>> I need some help using Hadoop :
>>
>> I'm trying to do some Dimensional reduction after this tutorial:
>> https://cwiki.apache.org/confluence/display/MAHOUT/Dimensional+Reduction
>>
>> I created my tf-idf-vectors from text saved in lucene:
>> https://cwiki.apache.org/confluence/display/MAHOUT/Creating+Vectors+from+Text
>>
>> Now, I got a problem when I try to transpose the tf-idf-vectors:
>> java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot
>> be cast to org.apache.hadoop.io.IntWritable
>>
>> So, my question:
>> What is the easiest way to get my data from
>> SequenceFile<LongWritable,VectorWritable> to
>> SequenceFile<IntWritable,VectorWritable>?
>> Or can I create this directly from a Lucene-Index?
>>
>> Thx
>> Stefan
>>
>
--
Stefan Wienert
http://www.wienert.cc
stefan@wienert.cc
Telefon: +495251-2026838 (neue Nummer seit 20.06.10)
Mobil: +49176-40170270
Re: How to convert SequenceFile to SequenceFile?
Posted by Sean Owen <sr...@gmail.com>.
(This wiki looks well out of date, recent as of only version 0.2. For
example we don't use IntWritable anymore. It needs an update, but,
ought not be too different as of now, version 0.5.)
You would need to key by int instead of long, conceptually. Do you
really have 32-bit values? then just write out an int in a Writable,
not long.
If not, you can just hash. I use elsewhere what Java uses: value ^
(value >>> 32). Actually I also add "& 0x7FFFFFFF" to make sure it's
positive.
You would need to store the reverse mapping from int to long to get
your original values out later. And there is a tiny chance of
collision.
On Wed, May 25, 2011 at 12:59 PM, Stefan Wienert <st...@wienert.cc> wrote:
> Hi,
>
> I need some help using Hadoop :
>
> I'm trying to do some Dimensional reduction after this tutorial:
> https://cwiki.apache.org/confluence/display/MAHOUT/Dimensional+Reduction
>
> I created my tf-idf-vectors from text saved in lucene:
> https://cwiki.apache.org/confluence/display/MAHOUT/Creating+Vectors+from+Text
>
> Now, I got a problem when I try to transpose the tf-idf-vectors:
> java.lang.ClassCastException: org.apache.hadoop.io.LongWritable cannot
> be cast to org.apache.hadoop.io.IntWritable
>
> So, my question:
> What is the easiest way to get my data from
> SequenceFile<LongWritable,VectorWritable> to
> SequenceFile<IntWritable,VectorWritable>?
> Or can I create this directly from a Lucene-Index?
>
> Thx
> Stefan
>