You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Robin Anil <ro...@gmail.com> on 2010/03/31 07:40:21 UTC
Re: Issues with memory use and inconsistent or state-influenced
results when using CBayesAlgorithm
On Tue, Mar 30, 2010 at 10:45 PM, Loek Cleophas
<lo...@kalooga.com>wrote:
> Hi,
>
> after my initial experiments with Mahout's Bayes and CBayes implementations
> on my company's dataset, we're now trying to integrate Mahout to classify
> our data in a production environment. We are however running into two odd
> issues, after having succesfuly trained a classifier (using CBayes).
>
> We're loading the trained model into an InMemoryBayesDataStore, and are
> able to get classification results (i.e. categories plus weights). However,
> we're seeing two odd issues:
>
> 1) it turns out the classifier's memory use increases by classifying a
> document; as a result, after a number of documents to classify, we run into
> memory issues.
> 2) somehow, classification is not consistent: e.g. if we classify text 1,
> 2, 3, and then 1 again, the second time text 1 is fed, we get slightly
> different weights - not by a lot, but not by little enough to discard it as
> floating point rounding issues; and if we classify text 1 and then 1 again
> without any intermediate classification on other texts, the weights do not
> change.
>
This shouldn't be happening. I mean there is nothing getting changed in
there.
>
> My colleagues and I have looked at the Mahout code, and it seems that the
> memory use increase is due to getLabelID in InMemoryBayesDatastore - which
> adds a label to a dictionary if it's not in there yet, but never seems to
> remove any labels from the dictionary. Could this be the source of the
> memory issue? I can imagine that if you're adding words that were not in the
> model but occur in text to be classified, this might increase memory use but
> probably shouldn't be happening (as it's classification, not training).
>
The number of labels is fixed right? 2,3 to 100s of a few thousands and not
more. I dont see why that should be a reason for alarm. If there is a leak
it could be because of something else.
>
> Any thoughts on these two issues, whether they're related, and what to do
> about them?
>
> Robin, I suspect/hope you're able to help here?
Can you tell me the model size in memory after first load, the dictionary
size, the label size, and the increment in memory usage after a
classification
>
> Regards,
> Loek
>
Robin
Re: Issues with memory use and inconsistent or state-influenced
results when using CBayesAlgorithm
Posted by Mut <aa...@gmail.com>.
Hi,
Thanks for your post. However, the proposed solution will not work because
the getFeatureID is needed to populated the weight matrix. So the proposed
modifications to the code will result in not loading the model correctly and
a wrong execution.
The problem with the large memory requirement comes from loading the model
itself with a double value to the weight matrix.
To fix it, we need to code to remove the features that will not affect the
classification (small weight) but still occupy memory.
I would appreciate any fix to the large amount of heap needed to run the
classification tasks.
Regards
--
View this message in context: http://lucene.472066.n3.nabble.com/Issues-with-memory-use-and-inconsistent-or-state-influenced-results-when-using-CBayesAlgorithm-tp686329p3258675.html
Sent from the Mahout User List mailing list archive at Nabble.com.
Re: Issues with memory use and inconsistent or state-influenced
results when using CBayesAlgorithm
Posted by Robin Anil <ro...@gmail.com>.
This will work for now. We can remove the addition features to the
dictionary altogether. Will yield better performance, and lock down the
model. Will require a bit more modification.
On Wed, Mar 31, 2010 at 7:59 PM, Ferdy <fe...@kalooga.com> wrote:
> Just solved it. You're right about resetting the datastore.
>
> During classification, every word still unknown is added to
> featureDictionary. This leads to the excessive growth if lots of texts with
> unknown words are to be classified. The inconsistency is caused by using a
> "vocabCount" that is not reset after each classification. Indeed,
> featureDictionary.size() is used for "vocabCount", which increases every
> time new unknown words are discovered.
>
> My suggestions for a patch:
> 1) Add to Datastore: *public void reset();*
> 2) Call this method in ClassifierContext, in both classifyDocument(...)
> methods. Should be the first statement.
> 3) Changes to InMemoryBayesDatastore:
> 3a) Add:
> private boolean keepState;
> private Set<String> temporaryNewFeatures;
> public void reset() {
> temporaryNewFeatures = new HashSet<String>();
> }
> 3b) Change line 119: *"return featureDictionary.size();* to *
> featureDictionary.size()+temporaryNewFeatures.size();*
> 3c) Change method getFeatureID to:
> private int getFeatureID(String feature) {
> if (featureDictionary.containsKey(feature)) {
> return featureDictionary.get(feature);
> } else {
> if (keepState) {
> temporaryNewFeatures.add(feature);
> return -1;
> }
> int id = featureDictionary.size();
> featureDictionary.put(feature, id);
> return id;
> }
> }
> 3d) Finally, add a final statement to initialize(): *keepState = true;*
>
> This worked for us. At least the consistency problem is solved. The first
> classified document returns the same result result as it would without these
> changes. Following classified documents have slightly different label scores
> (compared to before these changes), which make perfectly sense.
>
> I'm sorry for not making a patch myself, but I'm still having trouble
> setting up Mahout in Eclipse. I created completely new implementations by
> copy-pasting complete classes with small modifications.
>
>
>
> Sean Owen wrote:
>
> I'm no expert here but isn't it more likely the memory consumption is
> coming from the feature dictionary, and feature-label matrix? Because
> the label count indeed is fixed and small.
>
> This is a great job for a profiler like JProfiler -- would let you
> easily see where the heap is being consumed. A bit more manual, but
> free, is the jmap tool in Java:http://www.startux.de/index.php/java/45-java-heap-dumpyvComment45 A
> quick moment with this might easily demonstrate what's taking the
> memory.
>
> Resetting the data store might well be the right thing to do, if my
> guess is right.
>
>
>
Re: Issues with memory use and inconsistent or state-influenced
results when using CBayesAlgorithm
Posted by Ferdy <fe...@kalooga.com>.
Just solved it. You're right about resetting the datastore.
During classification, every word still unknown is added to
featureDictionary. This leads to the excessive growth if lots of texts
with unknown words are to be classified. The inconsistency is caused by
using a "vocabCount" that is not reset after each classification.
Indeed, featureDictionary.size() is used for "vocabCount", which
increases every time new unknown words are discovered.
My suggestions for a patch:
1) Add to Datastore: *public void reset();*
2) Call this method in ClassifierContext, in both classifyDocument(...)
methods. Should be the first statement.
3) Changes to InMemoryBayesDatastore:
3a) Add:
private boolean keepState;
private Set<String> temporaryNewFeatures;
public void reset() {
temporaryNewFeatures = new HashSet<String>();
}
3b) Change line 119: *"return featureDictionary.size();* to
*featureDictionary.size()+temporaryNewFeatures.size();*
3c) Change method getFeatureID to:
private int getFeatureID(String feature) {
if (featureDictionary.containsKey(feature)) {
return featureDictionary.get(feature);
} else {
if (keepState) {
temporaryNewFeatures.add(feature);
return -1;
}
int id = featureDictionary.size();
featureDictionary.put(feature, id);
return id;
}
}
3d) Finally, add a final statement to initialize(): *keepState = true;*
This worked for us. At least the consistency problem is solved. The
first classified document returns the same result result as it would
without these changes. Following classified documents have slightly
different label scores (compared to before these changes), which make
perfectly sense.
I'm sorry for not making a patch myself, but I'm still having trouble
setting up Mahout in Eclipse. I created completely new implementations
by copy-pasting complete classes with small modifications.
Sean Owen wrote:
> I'm no expert here but isn't it more likely the memory consumption is
> coming from the feature dictionary, and feature-label matrix? Because
> the label count indeed is fixed and small.
>
> This is a great job for a profiler like JProfiler -- would let you
> easily see where the heap is being consumed. A bit more manual, but
> free, is the jmap tool in Java:
> http://www.startux.de/index.php/java/45-java-heap-dumpyvComment45 A
> quick moment with this might easily demonstrate what's taking the
> memory.
>
> Resetting the data store might well be the right thing to do, if my
> guess is right.
>
Re: Re: Issues with memory use and inconsistent or state-influenced
results when using CBayesAlgorithm
Posted by Sean Owen <sr...@gmail.com>.
I'm no expert here but isn't it more likely the memory consumption is
coming from the feature dictionary, and feature-label matrix? Because
the label count indeed is fixed and small.
This is a great job for a profiler like JProfiler -- would let you
easily see where the heap is being consumed. A bit more manual, but
free, is the jmap tool in Java:
http://www.startux.de/index.php/java/45-java-heap-dumpyvComment45 A
quick moment with this might easily demonstrate what's taking the
memory.
Resetting the data store might well be the right thing to do, if my
guess is right.
Re: Re: Issues with memory use and inconsistent or state-influenced
results when using CBayesAlgorithm
Posted by Ferdy <fe...@kalooga.com>.
Hi,
Thanks for responding. I will fill in the details in addition to those
already provided by my colleague Loek. It applies to the Mahout release 0.3.
Robin Anil wrote:
> On Tue, Mar 30, 2010 at 10:45 PM, Loek Cleophas
> <lo...@kalooga.com>wrote:
>
>
>> Hi,
>>
>> after my initial experiments with Mahout's Bayes and CBayes implementations
>> on my company's dataset, we're now trying to integrate Mahout to classify
>> our data in a production environment. We are however running into two odd
>> issues, after having succesfuly trained a classifier (using CBayes).
>>
>> We're loading the trained model into an InMemoryBayesDataStore, and are
>> able to get classification results (i.e. categories plus weights). However,
>> we're seeing two odd issues:
>>
>> 1) it turns out the classifier's memory use increases by classifying a
>> document; as a result, after a number of documents to classify, we run into
>> memory issues.
>> 2) somehow, classification is not consistent: e.g. if we classify text 1,
>> 2, 3, and then 1 again, the second time text 1 is fed, we get slightly
>> different weights - not by a lot, but not by little enough to discard it as
>> floating point rounding issues; and if we classify text 1 and then 1 again
>> without any intermediate classification on other texts, the weights do not
>> change.
>>
>>
> This shouldn't be happening. I mean there is nothing getting changed in
> there.
>
I believe this not the case. The field featureDictionary in
InMemoryBayesDatastore is the cause of the severe memory growth and the
inconsistent classifying results. This is how I noticed it: In a small
testcase, I'll load a classifier and start feeding a few documents. The
Algorithm is CBayesAlgorithm and the Datastore is
InMemoryBayesDatastore. After the classifier is fully loaded, that is,
after initialize() is called, I start debugging the Mahout code. It
seems that 'features' are put into the featureDictionary DURING
classifying. Here's a stacktrace:
Thread [main] (Suspended (breakpoint at line 143 in
InMemoryBayesDatastore))
InMemoryBayesDatastore.getFeatureID(String) line: 143
InMemoryBayesDatastore.getWeight(String, String, String) line: 106
CBayesAlgorithm.featureWeight(Datastore, String, String) line: 93
CBayesAlgorithm$1.apply(String, int) line: 136
CBayesAlgorithm$1.apply(Object, int) line: 131
OpenObjectIntHashMap<T>.forEachPair(ObjectIntProcedure<T>) line: 186
CBayesAlgorithm.documentWeight(Datastore, String, String[]) line:
131
CBayesAlgorithm.classifyDocument(String[], Datastore, String, int)
line: 69
ClassifierContext.classifyDocument(String[], String, int) line: 81
Line 143 is a 'put' into the featureDictionary. Surely, not all features
contained in the documents are put, as some are already contained in it
(perhaps loaded during the initializing stage of the classifier).
Because the featureDictionary is used in the calcution of classifying,
results are not always inconsistent. The order of the documents and thus
the order of the first appearances of features seem to influence the
label scores for each document.
>
>> My colleagues and I have looked at the Mahout code, and it seems that the
>> memory use increase is due to getLabelID in InMemoryBayesDatastore - which
>> adds a label to a dictionary if it's not in there yet, but never seems to
>> remove any labels from the dictionary. Could this be the source of the
>> memory issue? I can imagine that if you're adding words that were not in the
>> model but occur in text to be classified, this might increase memory use but
>> probably shouldn't be happening (as it's classification, not training).
>>
>>
> The number of labels is fixed right? 2,3 to 100s of a few thousands and not
> more. I dont see why that should be a reason for alarm. If there is a leak
> it could be because of something else.
>
The number of labels is indeed fixed. Therefore, on second thought, the
labelDictionary is not really the problem. (At least not solely the
problem).
>
>> Any thoughts on these two issues, whether they're related, and what to do
>> about them?
>>
>> Robin, I suspect/hope you're able to help here?
>>
>
> Can you tell me the model size in memory after first load, the dictionary
> size, the label size, and the increment in memory usage after a
> classification
>
We trained a classifier using about 37000 input docs (10MB textual data)
and about 10 labels. The size of resulting classifier is 130MB on disk.
Loading the classifer in memory (after initializing) requires 600MB.
During classifying, the memory footprint slowly increases. The exact
growth after each document classification is not always the same and
difficult to measure accurately. Therefore I'll resort to totals.
The amount of input tokens per documents is the main factor in how many
documents we are ably to classify before a "GC overhead limit exceeded"
kicks in. In this particular case we're using a 1GB heapsize. Sometimes
we are able to classify 100000 documents, but with heavy documents the
limit can be as low as 20000 documents. To be certain we're dealing with
a memoryleak in Mahout, I reran the classifying code with a simple
change: Instead of using a single classifier, I reload the classifier
every 1000 documents (by means of creating a new InMemoryBayesDatastore
and discarding the old one). This allows the garbage collector to clean
the old InMemoryBayesDatastore. This change allows us the classify all
documents that would have previously been failed.
>> Regards,
>> Loek
>>
>
> Robin
>
Regards, Ferdy