You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Bernd Fehling <be...@uni-bielefeld.de> on 2011/06/22 16:14:45 UTC
size of synonyms.txt
While trying some synonyms.txt files I noticed a huge increase of heap usage.
synonyms_1.txt --> 6645 lines (2826104 bytes in size)
results in 66364 entries in SynonymMap with 730MB heap usage.
Startup time about 2 minutes.
synonyms_2.txt --> 6645 lines (5384884 bytes in size)
results in 115168 entries in SynonymMap with 3.3GB heap usage.
Startup time about 4 minutes.
What is your size of synonyms.txt?
Any limitations (e.g. file size, number of synonyms, ...)?
How to deal with _really_ large numbers of synonyms?
To the experts:
Why not using synonyms from a file, just because memory is faster?
Regards,
Bernd
Re: size of synonyms.txt
Posted by Bernd Fehling <be...@uni-bielefeld.de>.
> On Wed, Jun 22, 2011 at 10:14 AM, Bernd Fehling
> <be...@uni-bielefeld.de> wrote:
> > While trying some synonyms.txt files I noticed a huge increase
> of heap
> > usage.
> >
> > synonyms_1.txt --> 6645 lines (2826104 bytes in size)
> > results in 66364 entries in SynonymMap with 730MB heap usage.
> > Startup time about 2 minutes.
> >
> > synonyms_2.txt --> 6645 lines (5384884 bytes in size)
> > results in 115168 entries in SynonymMap with 3.3GB heap usage.
> > Startup time about 4 minutes.
> >
> >
> > What is your size of synonyms.txt?
> >
> >
> > Any limitations (e.g. file size, number of synonyms, ...)?
> >
> >
> > How to deal with _really_ large numbers of synonyms?
> >
> >
> > To the experts:
> > Why not using synonyms from a file, just because memory is faster?
> >
>
> Hi,
>
> I think we should look at implementing synonyms with an FST, to reduce
> the ram usage.
> I also think this would make it easier for us to minimize the number
> of captureState/restoreState that it does,
> because it would just be a more natural way to handle all the
> multi-word cases... this could actually speed up the analysis
> time for
> this filter.
Wow you can read between the lines ;-)
Exactly what I have on my mind.
Re: size of synonyms.txt
Posted by Darren Govoni <da...@ontrenet.com>.
I once tried to load wordnet synsets as a synonym file and it was
prohibitively slow and unusable. fyi.
On 06/22/2011 12:23 PM, Robert Muir wrote:
> On Wed, Jun 22, 2011 at 10:14 AM, Bernd Fehling
> <be...@uni-bielefeld.de> wrote:
>> While trying some synonyms.txt files I noticed a huge increase of heap
>> usage.
>>
>> synonyms_1.txt --> 6645 lines (2826104 bytes in size)
>> results in 66364 entries in SynonymMap with 730MB heap usage.
>> Startup time about 2 minutes.
>>
>> synonyms_2.txt --> 6645 lines (5384884 bytes in size)
>> results in 115168 entries in SynonymMap with 3.3GB heap usage.
>> Startup time about 4 minutes.
>>
>>
>> What is your size of synonyms.txt?
>>
>>
>> Any limitations (e.g. file size, number of synonyms, ...)?
>>
>>
>> How to deal with _really_ large numbers of synonyms?
>>
>>
>> To the experts:
>> Why not using synonyms from a file, just because memory is faster?
>>
> Hi,
>
> I think we should look at implementing synonyms with an FST, to reduce
> the ram usage.
> I also think this would make it easier for us to minimize the number
> of captureState/restoreState that it does,
> because it would just be a more natural way to handle all the
> multi-word cases... this could actually speed up the analysis time for
> this filter.
Re: size of synonyms.txt
Posted by Robert Muir <rc...@gmail.com>.
On Wed, Jun 22, 2011 at 10:14 AM, Bernd Fehling
<be...@uni-bielefeld.de> wrote:
> While trying some synonyms.txt files I noticed a huge increase of heap
> usage.
>
> synonyms_1.txt --> 6645 lines (2826104 bytes in size)
> results in 66364 entries in SynonymMap with 730MB heap usage.
> Startup time about 2 minutes.
>
> synonyms_2.txt --> 6645 lines (5384884 bytes in size)
> results in 115168 entries in SynonymMap with 3.3GB heap usage.
> Startup time about 4 minutes.
>
>
> What is your size of synonyms.txt?
>
>
> Any limitations (e.g. file size, number of synonyms, ...)?
>
>
> How to deal with _really_ large numbers of synonyms?
>
>
> To the experts:
> Why not using synonyms from a file, just because memory is faster?
>
Hi,
I think we should look at implementing synonyms with an FST, to reduce
the ram usage.
I also think this would make it easier for us to minimize the number
of captureState/restoreState that it does,
because it would just be a more natural way to handle all the
multi-word cases... this could actually speed up the analysis time for
this filter.