Vector creation - out of memory error

[Florian Leibert <fl...@leibert.de>]

Hi,
I'm trying to create vectors with Mahout as explained in
http://cwiki.apache.org/confluence/display/MAHOUT/Creating+Vectors+from+Text,
however I keep running out of heap. My heap is set to 2 GB already and I use
these parameters:
"java org.apache.mahout.utils.vectors.Driver --dir /LUCENE/ind --output
/user/florian/index-vectors-01 --field content --dictOut
/user/florian/index-dict-01 --weight TF".

My index currently is about 6 GB large. Is there any way to compute the
vectors in a distributed manner? What's the largest index someone has
created vectors from?

Thanks!

Florian

Re: Vector creation - out of memory error

[Florian Leibert <fl...@leibert.de>]

Hi Grant,
thanks for your answers - it seems to work with a heap of 4GB - but is
fairly slow. I'd be interested in seeing if we could make this process
distributed? It's running as a standalone right now and thus is a
bottleneck...

Are there any attempts right now to implement it in a M/R fashion?

Thanks,
Florian

On Mon, Jul 20, 2009 at 5:49 PM, Grant Ingersoll <gs...@apache.org>wrote:

>
>
>
> On Jul 20, 2009, at 2:40 PM, Florian Leibert wrote:
>
>  Hi,
>> I'm trying to create vectors with Mahout as explained in
>>
>> http://cwiki.apache.org/confluence/display/MAHOUT/Creating+Vectors+from+Text
>> ,
>> however I keep running out of heap. My heap is set to 2 GB already and I
>> use
>> these parameters:
>> "java org.apache.mahout.utils.vectors.Driver --dir /LUCENE/ind --output
>> /user/florian/index-vectors-01 --field content --dictOut
>> /user/florian/index-dict-01 --weight TF".
>>
>
> Hmm, 6GB isn't all that large, but the primary memory usage is going to be
> due to the CachedTermInfo, which loads all the terms into memory.  This is
> an interface that can be implemented in other, slower, ways, but we'll have
> to change the Driver program to allow for that.
>
> How many unique terms do you have in the content field?
>
> You have java -Xmx2000M set as the heap size?
>
>
>
>> My index currently is about 6 GB large. Is there any way to compute the
>> vectors in a distributed manner?
>>
>
> There will be, but there isn't yet, I suspect.
>
>
>  What's the largest index someone has
>> created vectors from?
>>
>
> It's pretty new code, I've only tested it on relatively small indexes (few
> 100 mgs) so far, but the only gating issue memory wise is the
> CachedTermInfo.
>
> Sorry I don't have better answers, but I am willing to help improve.  I
> will try to use some bigger indexes soon.
>
> -Grant
>

Re: Vector creation - out of memory error

[Grant Ingersoll <gs...@apache.org>]

On Jul 20, 2009, at 2:40 PM, Florian Leibert wrote:

> Hi,
> I'm trying to create vectors with Mahout as explained in
> http://cwiki.apache.org/confluence/display/MAHOUT/Creating+Vectors+from+Text 
> ,
> however I keep running out of heap. My heap is set to 2 GB already  
> and I use
> these parameters:
> "java org.apache.mahout.utils.vectors.Driver --dir /LUCENE/ind -- 
> output
> /user/florian/index-vectors-01 --field content --dictOut
> /user/florian/index-dict-01 --weight TF".

Hmm, 6GB isn't all that large, but the primary memory usage is going  
to be due to the CachedTermInfo, which loads all the terms into  
memory.  This is an interface that can be implemented in other,  
slower, ways, but we'll have to change the Driver program to allow for  
that.

How many unique terms do you have in the content field?

You have java -Xmx2000M set as the heap size?


>
> My index currently is about 6 GB large. Is there any way to compute  
> the
> vectors in a distributed manner?

There will be, but there isn't yet, I suspect.


> What's the largest index someone has
> created vectors from?

It's pretty new code, I've only tested it on relatively small indexes  
(few 100 mgs) so far, but the only gating issue memory wise is the  
CachedTermInfo.

Sorry I don't have better answers, but I am willing to help improve.   
I will try to use some bigger indexes soon.

-Grant

Re: Vector creation - out of memory error

[Shashikant Kore <sh...@gmail.com>]

My understanding of MR is embarassingly low, but I am unable to see
how it can drastically improve the performance. Each node in Hadoop
cluster has to load the document frequencies consuming same amount of
memory. They may operate on different document ranges, giving linear
performance improvement.

Here are couple of thoughts. See if these are sensible.

a) Vector generation process is serial right now. During each pass
some time is spent to retrive term vectors (which incurs disk access),
some to calculate weights (cpu intensive) and finally writing vector
top disk (again disk access.)  If the process is multi-threaded, there
will be a nice pipeline which keeps cpu and disk busy all the time. I
suspect, disk will hit saturation before cpu.

b)  I am not sure if you are storing term positions in the index. To
calculate vectors you don't require term positions. If you only store
term frequency during indexing, index size will come down quite a bit.
In which case, you can load the entire index in memory.  Disk access
is completely eliminated and vector generation will be fast. But
again, this is assuming you don't need term vectors.

As Grant has pointed out JSON is not useful for clustering. You may
switch it off.

--shashi

On Tue, Jul 21, 2009 at 11:44 PM, Florian Leibert<fl...@leibert.de> wrote:
> Hi Shashi,
> great - I'm trying the settings maxDFPercent 50 and minDF 4 - I have a lot
> of very short documents of which some can be very descriptive.
> I'm thinking I should have used the StopWordAnalyzer in Lucene when creating
> the index - that way the creation of the vectors would be much faster.
>
> It took yesterday about 8 hours to process these vectors on a quad core
> machine with 4 GB of heap - using the sequence file writer - I assume that
> the bottleneck might have been the constant transfer into HDFS - that's why
> I'm using the file writer now. It's running on my 6 GB index since about 90
> minutes now and while the vector sequence file yesterday was 3 GB large
> (without filtering) - the JSON file is already at 16 GB (with filtering) -
> which I attribute to the compression of the sequence file...
>
> I'm trying to allot some time to transform the vector creation process to
> M/R if nobody else is working on that at the moment...
>
> Florian
>
>
> On Mon, Jul 20, 2009 at 10:46 PM, Shashikant Kore <sh...@gmail.com>wrote:
>
>> You can restrict the term set by applying "minDf"  & "maxDFPercent"
>> filters.
>>
>> Idea behind the parameters is that the terms occurring too frequently
>> or too rarely are not very useful. If you set "minDf" parameter to 10,
>> the term has to appear in at least 10 documents in the index.
>> Similarly, if "maxDFPercent" is set to 50, all terms appearing in more
>> than 50% documents are ignored.
>>
>> These two parameters prune the term set drastically. I wouldn't be
>> suprised if the term set shrinks to less 10% of the original set.
>> Since, the vector generation code keeps term->doc-freq map in memory,
>> the memory footprint is now at a "manageable" level. Also, vector
>> generation will be faster as there are fewer features features per
>> vector.
>>
>> BTW, how slow is vector generation? I don't have exact figures with
>> me, but on a single box, I recall it to be higher than 50 vectors per
>> second.
>>
>> --shashi
>>
>> On Tue, Jul 21, 2009 at 12:10 AM, Florian Leibert<fl...@leibert.de> wrote:
>> > Hi,
>> > I'm trying to create vectors with Mahout as explained in
>> >
>> http://cwiki.apache.org/confluence/display/MAHOUT/Creating+Vectors+from+Text
>> ,
>> > however I keep running out of heap. My heap is set to 2 GB already and I
>> use
>> > these parameters:
>> > "java org.apache.mahout.utils.vectors.Driver --dir /LUCENE/ind --output
>> > /user/florian/index-vectors-01 --field content --dictOut
>> > /user/florian/index-dict-01 --weight TF".
>> >
>> > My index currently is about 6 GB large. Is there any way to compute the
>> > vectors in a distributed manner? What's the largest index someone has
>> > created vectors from?
>> >
>> > Thanks!
>> >
>> > Florian
>> >
>>

Re: Vector creation - out of memory error

[Grant Ingersoll <gs...@apache.org>]

On Jul 21, 2009, at 2:14 PM, Florian Leibert wrote:

> Hi Shashi,
> great - I'm trying the settings maxDFPercent 50 and minDF 4 - I have  
> a lot
> of very short documents of which some can be very descriptive.
> I'm thinking I should have used the StopWordAnalyzer in Lucene when  
> creating
> the index - that way the creation of the vectors would be much faster.
>
> It took yesterday about 8 hours to process these vectors on a quad  
> core
> machine with 4 GB of heap - using the sequence file writer - I  
> assume that
> the bottleneck might have been the constant transfer into HDFS

You might try writing out to a local path and then simply copying to  
HDFS once it is done.

> - that's why
> I'm using the file writer now. It's running on my 6 GB index since  
> about 90
> minutes now and while the vector sequence file yesterday was 3 GB  
> large
> (without filtering) - the JSON file is already at 16 GB (with  
> filtering) -
> which I attribute to the compression of the sequence file...

The JSON file isn't going to do much for you in terms of actually  
clustering, if that is what you are after.  The clustering algorithms  
work on SequenceFiles only.  The JSON stuff is only really useful for  
human consumption, I guess.


>
> I'm trying to allot some time to transform the vector creation  
> process to
> M/R if nobody else is working on that at the moment...
>

That would be great, but likely somewhat tricky with the Lucene  
index.  Note, also that the current approach is just one approach for  
creating the matrix.  You certainly don't have to go through Lucene.   
You could implement other types of VectorIterables, or just forgo that  
all together and create the sequence file on you own.

-Grant

Re: Vector creation - out of memory error

[Florian Leibert <fl...@leibert.de>]

Hi Shashi,
great - I'm trying the settings maxDFPercent 50 and minDF 4 - I have a lot
of very short documents of which some can be very descriptive.
I'm thinking I should have used the StopWordAnalyzer in Lucene when creating
the index - that way the creation of the vectors would be much faster.

It took yesterday about 8 hours to process these vectors on a quad core
machine with 4 GB of heap - using the sequence file writer - I assume that
the bottleneck might have been the constant transfer into HDFS - that's why
I'm using the file writer now. It's running on my 6 GB index since about 90
minutes now and while the vector sequence file yesterday was 3 GB large
(without filtering) - the JSON file is already at 16 GB (with filtering) -
which I attribute to the compression of the sequence file...

I'm trying to allot some time to transform the vector creation process to
M/R if nobody else is working on that at the moment...

Florian


On Mon, Jul 20, 2009 at 10:46 PM, Shashikant Kore <sh...@gmail.com>wrote:

> You can restrict the term set by applying "minDf"  & "maxDFPercent"
> filters.
>
> Idea behind the parameters is that the terms occurring too frequently
> or too rarely are not very useful. If you set "minDf" parameter to 10,
> the term has to appear in at least 10 documents in the index.
> Similarly, if "maxDFPercent" is set to 50, all terms appearing in more
> than 50% documents are ignored.
>
> These two parameters prune the term set drastically. I wouldn't be
> suprised if the term set shrinks to less 10% of the original set.
> Since, the vector generation code keeps term->doc-freq map in memory,
> the memory footprint is now at a "manageable" level. Also, vector
> generation will be faster as there are fewer features features per
> vector.
>
> BTW, how slow is vector generation? I don't have exact figures with
> me, but on a single box, I recall it to be higher than 50 vectors per
> second.
>
> --shashi
>
> On Tue, Jul 21, 2009 at 12:10 AM, Florian Leibert<fl...@leibert.de> wrote:
> > Hi,
> > I'm trying to create vectors with Mahout as explained in
> >
> http://cwiki.apache.org/confluence/display/MAHOUT/Creating+Vectors+from+Text
> ,
> > however I keep running out of heap. My heap is set to 2 GB already and I
> use
> > these parameters:
> > "java org.apache.mahout.utils.vectors.Driver --dir /LUCENE/ind --output
> > /user/florian/index-vectors-01 --field content --dictOut
> > /user/florian/index-dict-01 --weight TF".
> >
> > My index currently is about 6 GB large. Is there any way to compute the
> > vectors in a distributed manner? What's the largest index someone has
> > created vectors from?
> >
> > Thanks!
> >
> > Florian
> >
>

Re: Vector creation - out of memory error

[Shashikant Kore <sh...@gmail.com>]

You can restrict the term set by applying "minDf"  & "maxDFPercent" filters.

Idea behind the parameters is that the terms occurring too frequently
or too rarely are not very useful. If you set "minDf" parameter to 10,
the term has to appear in at least 10 documents in the index.
Similarly, if "maxDFPercent" is set to 50, all terms appearing in more
than 50% documents are ignored.

These two parameters prune the term set drastically. I wouldn't be
suprised if the term set shrinks to less 10% of the original set.
Since, the vector generation code keeps term->doc-freq map in memory,
the memory footprint is now at a "manageable" level. Also, vector
generation will be faster as there are fewer features features per
vector.

BTW, how slow is vector generation? I don't have exact figures with
me, but on a single box, I recall it to be higher than 50 vectors per
second.

--shashi

On Tue, Jul 21, 2009 at 12:10 AM, Florian Leibert<fl...@leibert.de> wrote:
> Hi,
> I'm trying to create vectors with Mahout as explained in
> http://cwiki.apache.org/confluence/display/MAHOUT/Creating+Vectors+from+Text,
> however I keep running out of heap. My heap is set to 2 GB already and I use
> these parameters:
> "java org.apache.mahout.utils.vectors.Driver --dir /LUCENE/ind --output
> /user/florian/index-vectors-01 --field content --dictOut
> /user/florian/index-dict-01 --weight TF".
>
> My index currently is about 6 GB large. Is there any way to compute the
> vectors in a distributed manner? What's the largest index someone has
> created vectors from?
>
> Thanks!
>
> Florian
>