You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Matteo Fiandesio <ma...@gmail.com> on 2010/06/18 16:52:49 UTC
Re: OOM on sorting on dynamic fields
Hello,
we are experiencing OOM exceptions in our single core solr instance
(on a (huge) amazon EC2 machine).
We investigated a lot in the mailing list and through jmap/jhat dump
analyzing and the problem resides in the lucene FieldCache that fills
the heap and blows up the server.
Our index is quite small but we have a lot of sort queries on fields
that are dynamic,of type long representing timestamps and are not
present in all the documents.
Those queries apply sorting on 12-15 of those fields.
We are using solr 1.4 in production and the dump shows a lot of
Integer/Character and Byte Array filled up with 0s.
With solr's trunk code things does not change.
In the mailing list we saw a lot of messages related to this issues:
we tried truncating the dates to day precision,using missingSortLast =
true,changing the field type from slong to long,setting autowarming to
different values,disabling and enabling caches with different values
but we did not manage to solve the problem.
We were thinking to implement an LRUFieldCache field type to manage
the FieldCache as an LRU and preventing but, before starting a new
development, we want to be sure that we are not doing anything wrong
in the solr configuration or in the index generation.
Any help would be appreciated.
Regards,
Matteo
Re: OOM on sorting on dynamic fields
Posted by Matteo Fiandesio <ma...@gmail.com>.
First of all thanks for your answers.
Those OOMEs are pretty nasty for our production environment.
I didn't try the solution of ordering by function as it was a solr 1.5
feature and we prefer to use a stable version 1.4.
I made a temporary patch that it looks is working fine.
I patched the lucene-core-2.9.1 source code adding those this lines in the
abstract static class Cache's get method
...
public Object get(IndexReader reader, Entry key) throws IOException {
Map innerCache;
Object value;
+ final Object readerKey = reader.getFieldCacheKey();
+ CacheEntry[] cacheEntries = wrapper.getCacheEntries();
+ if(cacheEntries.length>A_TUNED_INT_VALUE){
+ readerCache.clear();
+ }
...
I didn't notice any delay or concurrence problem.
On 22 June 2010 07:27, Lance Norskog <go...@gmail.com> wrote:
> No, this is basic to how Lucene works. You will need larger EC2 instances.
>
> On Mon, Jun 21, 2010 at 2:08 AM, Matteo Fiandesio
> <ma...@gmail.com> wrote:
>> Compiling solr with lucene 2.9.3 instead of 2.9.1 will solve this issue?
>> Regards,
>> Matteo
>>
>> On 19 June 2010 02:28, Lance Norskog <go...@gmail.com> wrote:
>>> The Lucene implementation of sorting creates an array of four-byte
>>> ints for every document in the index, and another array of the unique
>>> values in the field.
>>> If the timestamps are 'date' or 'tdate' in the schema, they do not
>>> need the second array.
>>>
>>> You can also sort by a field's with a function query. This does not
>>> build the arrays, but might be a little slower.
>>> Yes, the sort arrays (and also facet values for a field) should be
>>> controlled by a fixed-size cache, but they are not.
>>>
>>> On Fri, Jun 18, 2010 at 7:52 AM, Matteo Fiandesio
>>> <ma...@gmail.com> wrote:
>>>> Hello,
>>>> we are experiencing OOM exceptions in our single core solr instance
>>>> (on a (huge) amazon EC2 machine).
>>>> We investigated a lot in the mailing list and through jmap/jhat dump
>>>> analyzing and the problem resides in the lucene FieldCache that fills
>>>> the heap and blows up the server.
>>>>
>>>> Our index is quite small but we have a lot of sort queries on fields
>>>> that are dynamic,of type long representing timestamps and are not
>>>> present in all the documents.
>>>> Those queries apply sorting on 12-15 of those fields.
>>>>
>>>> We are using solr 1.4 in production and the dump shows a lot of
>>>> Integer/Character and Byte Array filled up with 0s.
>>>> With solr's trunk code things does not change.
>>>>
>>>> In the mailing list we saw a lot of messages related to this issues:
>>>> we tried truncating the dates to day precision,using missingSortLast =
>>>> true,changing the field type from slong to long,setting autowarming to
>>>> different values,disabling and enabling caches with different values
>>>> but we did not manage to solve the problem.
>>>>
>>>> We were thinking to implement an LRUFieldCache field type to manage
>>>> the FieldCache as an LRU and preventing but, before starting a new
>>>> development, we want to be sure that we are not doing anything wrong
>>>> in the solr configuration or in the index generation.
>>>>
>>>> Any help would be appreciated.
>>>> Regards,
>>>> Matteo
>>>>
>>>
>>>
>>>
>>> --
>>> Lance Norskog
>>> goksron@gmail.com
>>>
>>
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>
Re: OOM on sorting on dynamic fields
Posted by Lance Norskog <go...@gmail.com>.
No, this is basic to how Lucene works. You will need larger EC2 instances.
On Mon, Jun 21, 2010 at 2:08 AM, Matteo Fiandesio
<ma...@gmail.com> wrote:
> Compiling solr with lucene 2.9.3 instead of 2.9.1 will solve this issue?
> Regards,
> Matteo
>
> On 19 June 2010 02:28, Lance Norskog <go...@gmail.com> wrote:
>> The Lucene implementation of sorting creates an array of four-byte
>> ints for every document in the index, and another array of the unique
>> values in the field.
>> If the timestamps are 'date' or 'tdate' in the schema, they do not
>> need the second array.
>>
>> You can also sort by a field's with a function query. This does not
>> build the arrays, but might be a little slower.
>> Yes, the sort arrays (and also facet values for a field) should be
>> controlled by a fixed-size cache, but they are not.
>>
>> On Fri, Jun 18, 2010 at 7:52 AM, Matteo Fiandesio
>> <ma...@gmail.com> wrote:
>>> Hello,
>>> we are experiencing OOM exceptions in our single core solr instance
>>> (on a (huge) amazon EC2 machine).
>>> We investigated a lot in the mailing list and through jmap/jhat dump
>>> analyzing and the problem resides in the lucene FieldCache that fills
>>> the heap and blows up the server.
>>>
>>> Our index is quite small but we have a lot of sort queries on fields
>>> that are dynamic,of type long representing timestamps and are not
>>> present in all the documents.
>>> Those queries apply sorting on 12-15 of those fields.
>>>
>>> We are using solr 1.4 in production and the dump shows a lot of
>>> Integer/Character and Byte Array filled up with 0s.
>>> With solr's trunk code things does not change.
>>>
>>> In the mailing list we saw a lot of messages related to this issues:
>>> we tried truncating the dates to day precision,using missingSortLast =
>>> true,changing the field type from slong to long,setting autowarming to
>>> different values,disabling and enabling caches with different values
>>> but we did not manage to solve the problem.
>>>
>>> We were thinking to implement an LRUFieldCache field type to manage
>>> the FieldCache as an LRU and preventing but, before starting a new
>>> development, we want to be sure that we are not doing anything wrong
>>> in the solr configuration or in the index generation.
>>>
>>> Any help would be appreciated.
>>> Regards,
>>> Matteo
>>>
>>
>>
>>
>> --
>> Lance Norskog
>> goksron@gmail.com
>>
>
--
Lance Norskog
goksron@gmail.com
Re: OOM on sorting on dynamic fields
Posted by Matteo Fiandesio <ma...@gmail.com>.
Compiling solr with lucene 2.9.3 instead of 2.9.1 will solve this issue?
Regards,
Matteo
On 19 June 2010 02:28, Lance Norskog <go...@gmail.com> wrote:
> The Lucene implementation of sorting creates an array of four-byte
> ints for every document in the index, and another array of the unique
> values in the field.
> If the timestamps are 'date' or 'tdate' in the schema, they do not
> need the second array.
>
> You can also sort by a field's with a function query. This does not
> build the arrays, but might be a little slower.
> Yes, the sort arrays (and also facet values for a field) should be
> controlled by a fixed-size cache, but they are not.
>
> On Fri, Jun 18, 2010 at 7:52 AM, Matteo Fiandesio
> <ma...@gmail.com> wrote:
>> Hello,
>> we are experiencing OOM exceptions in our single core solr instance
>> (on a (huge) amazon EC2 machine).
>> We investigated a lot in the mailing list and through jmap/jhat dump
>> analyzing and the problem resides in the lucene FieldCache that fills
>> the heap and blows up the server.
>>
>> Our index is quite small but we have a lot of sort queries on fields
>> that are dynamic,of type long representing timestamps and are not
>> present in all the documents.
>> Those queries apply sorting on 12-15 of those fields.
>>
>> We are using solr 1.4 in production and the dump shows a lot of
>> Integer/Character and Byte Array filled up with 0s.
>> With solr's trunk code things does not change.
>>
>> In the mailing list we saw a lot of messages related to this issues:
>> we tried truncating the dates to day precision,using missingSortLast =
>> true,changing the field type from slong to long,setting autowarming to
>> different values,disabling and enabling caches with different values
>> but we did not manage to solve the problem.
>>
>> We were thinking to implement an LRUFieldCache field type to manage
>> the FieldCache as an LRU and preventing but, before starting a new
>> development, we want to be sure that we are not doing anything wrong
>> in the solr configuration or in the index generation.
>>
>> Any help would be appreciated.
>> Regards,
>> Matteo
>>
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>
Re: OOM on sorting on dynamic fields
Posted by Lance Norskog <go...@gmail.com>.
The Lucene implementation of sorting creates an array of four-byte
ints for every document in the index, and another array of the unique
values in the field.
If the timestamps are 'date' or 'tdate' in the schema, they do not
need the second array.
You can also sort by a field's with a function query. This does not
build the arrays, but might be a little slower.
Yes, the sort arrays (and also facet values for a field) should be
controlled by a fixed-size cache, but they are not.
On Fri, Jun 18, 2010 at 7:52 AM, Matteo Fiandesio
<ma...@gmail.com> wrote:
> Hello,
> we are experiencing OOM exceptions in our single core solr instance
> (on a (huge) amazon EC2 machine).
> We investigated a lot in the mailing list and through jmap/jhat dump
> analyzing and the problem resides in the lucene FieldCache that fills
> the heap and blows up the server.
>
> Our index is quite small but we have a lot of sort queries on fields
> that are dynamic,of type long representing timestamps and are not
> present in all the documents.
> Those queries apply sorting on 12-15 of those fields.
>
> We are using solr 1.4 in production and the dump shows a lot of
> Integer/Character and Byte Array filled up with 0s.
> With solr's trunk code things does not change.
>
> In the mailing list we saw a lot of messages related to this issues:
> we tried truncating the dates to day precision,using missingSortLast =
> true,changing the field type from slong to long,setting autowarming to
> different values,disabling and enabling caches with different values
> but we did not manage to solve the problem.
>
> We were thinking to implement an LRUFieldCache field type to manage
> the FieldCache as an LRU and preventing but, before starting a new
> development, we want to be sure that we are not doing anything wrong
> in the solr configuration or in the index generation.
>
> Any help would be appreciated.
> Regards,
> Matteo
>
--
Lance Norskog
goksron@gmail.com
Re: OOM on sorting on dynamic fields
Posted by Matteo Fiandesio <ma...@gmail.com>.
Hi to all,
we moved solr with patched lucene's FieldCache in production environment.
During tests we noticed random ConcurrentModificationException calling
the getCacheEntries method due to this bug
https://issues.apache.org/jira/browse/LUCENE-2273
We applied that patch as well, and added an abstract int
getCacheSize() method to FieldCache abstract class and its
implementation in abstract Cache inner class in CacheFieldImpl that
returns the cache size without instantiating a CacheEntry array.
Response time are slower on cache purging but acceptable from the user
point of view.
Regards,
Matteo
On 22 June 2010 22:41, Matteo Fiandesio <ma...@gmail.com> wrote:
> Fields over i'm sorting to are dynamic so one query sorts on
> erick_time_1,erick_timeA_1 and other sorts on erick_time_2 and so
> on.What we see in the heap are a lot of arrays,most of them,filled
> with 0s maybe due to the fact that this timestamps fields are not
> present in all the documents.
>
> By the way,
> I have a script that generates the OOM in 10 minutes on our solr
> instance and with the temporary patch it runned without any problems.
> The side effect is that when the cache is purged next query that
> regenerates the cache is a little bit slower.
>
> I'm aware that the solution is unelegant and we are investigating to
> solve the problem in another way.
> Regards,
> Matteo
>
>
> On 22 June 2010 19:25, Erick Erickson <er...@gmail.com> wrote:
>> Hmmm, I'm missing something here then. Sorting over 15 fields of type long
>> shouldn't use much memory, even if all the values are unique. When you say
>> "12-15 dynamic fields", are you talking about 12-15 fields per query out of
>> XXX total fields? And is XXX large? At a guess, how many different fields
>> do
>> you think you're sorting over cumulative by the time you get your OOM?
>> Note if you sort over the field "erick_time" in 10 different queries, I'm
>> only counting that as 1 field. I guess another way of asking this is
>> "how many dynamic fields are there total?".
>>
>> If this is really a sorting issue, you should be able to force this to
>> happen
>> almost immediately by firing off enough sort queries at the server. It'll
>> tell you a lot if you can't make this happen, even on a relatively small
>> test machine.
>>
>> Best
>> Erick
>>
>> On Tue, Jun 22, 2010 at 12:59 PM, Matteo Fiandesio <
>> matteo.fiandesio@gmail.com> wrote:
>>
>>> Hi Erick,
>>> the index is quite small (1691145 docs) but sorting is massive and
>>> often on unique timestamp fields.
>>>
>>> OOM occur after a range of time between three and four hours.
>>> Depending as well if users browse a part of the application.
>>>
>>> We use solrj to make the queries so we did not use Readers objects
>>> directly.
>>>
>>> Without sorting we don't see the problem
>>> Regards,
>>> Matteo
>>>
>>> On 22 June 2010 17:01, Erick Erickson <er...@gmail.com> wrote:
>>> > Hmmmm.. A couple of details I'm wondering about. How many
>>> > documents are we talking about in your index? Do you get
>>> > OOMs when you start fresh or does it take a while?
>>> >
>>> > You've done some good investigations, so it seems like there
>>> > could well be something else going on here than just "the usual
>>> > suspects" of sorting....
>>> >
>>> > I'm wondering if you aren't really closing readers somehow.
>>> > Are you updating your index frequently and re-opening readers often?
>>> > If so, how?
>>> >
>>> > I'm assuming that if you do NOT sort on all these fields, you don't have
>>> > the problem, is that true?
>>> >
>>> > Best
>>> > Erick
>>> >
>>> > On Fri, Jun 18, 2010 at 10:52 AM, Matteo Fiandesio <
>>> > matteo.fiandesio@gmail.com> wrote:
>>> >
>>> >> Hello,
>>> >> we are experiencing OOM exceptions in our single core solr instance
>>> >> (on a (huge) amazon EC2 machine).
>>> >> We investigated a lot in the mailing list and through jmap/jhat dump
>>> >> analyzing and the problem resides in the lucene FieldCache that fills
>>> >> the heap and blows up the server.
>>> >>
>>> >> Our index is quite small but we have a lot of sort queries on fields
>>> >> that are dynamic,of type long representing timestamps and are not
>>> >> present in all the documents.
>>> >> Those queries apply sorting on 12-15 of those fields.
>>> >>
>>> >> We are using solr 1.4 in production and the dump shows a lot of
>>> >> Integer/Character and Byte Array filled up with 0s.
>>> >> With solr's trunk code things does not change.
>>> >>
>>> >> In the mailing list we saw a lot of messages related to this issues:
>>> >> we tried truncating the dates to day precision,using missingSortLast =
>>> >> true,changing the field type from slong to long,setting autowarming to
>>> >> different values,disabling and enabling caches with different values
>>> >> but we did not manage to solve the problem.
>>> >>
>>> >> We were thinking to implement an LRUFieldCache field type to manage
>>> >> the FieldCache as an LRU and preventing but, before starting a new
>>> >> development, we want to be sure that we are not doing anything wrong
>>> >> in the solr configuration or in the index generation.
>>> >>
>>> >> Any help would be appreciated.
>>> >> Regards,
>>> >> Matteo
>>> >>
>>> >
>>>
>>
>
Re: OOM on sorting on dynamic fields
Posted by Matteo Fiandesio <ma...@gmail.com>.
Fields over i'm sorting to are dynamic so one query sorts on
erick_time_1,erick_timeA_1 and other sorts on erick_time_2 and so
on.What we see in the heap are a lot of arrays,most of them,filled
with 0s maybe due to the fact that this timestamps fields are not
present in all the documents.
By the way,
I have a script that generates the OOM in 10 minutes on our solr
instance and with the temporary patch it runned without any problems.
The side effect is that when the cache is purged next query that
regenerates the cache is a little bit slower.
I'm aware that the solution is unelegant and we are investigating to
solve the problem in another way.
Regards,
Matteo
On 22 June 2010 19:25, Erick Erickson <er...@gmail.com> wrote:
> Hmmm, I'm missing something here then. Sorting over 15 fields of type long
> shouldn't use much memory, even if all the values are unique. When you say
> "12-15 dynamic fields", are you talking about 12-15 fields per query out of
> XXX total fields? And is XXX large? At a guess, how many different fields
> do
> you think you're sorting over cumulative by the time you get your OOM?
> Note if you sort over the field "erick_time" in 10 different queries, I'm
> only counting that as 1 field. I guess another way of asking this is
> "how many dynamic fields are there total?".
>
> If this is really a sorting issue, you should be able to force this to
> happen
> almost immediately by firing off enough sort queries at the server. It'll
> tell you a lot if you can't make this happen, even on a relatively small
> test machine.
>
> Best
> Erick
>
> On Tue, Jun 22, 2010 at 12:59 PM, Matteo Fiandesio <
> matteo.fiandesio@gmail.com> wrote:
>
>> Hi Erick,
>> the index is quite small (1691145 docs) but sorting is massive and
>> often on unique timestamp fields.
>>
>> OOM occur after a range of time between three and four hours.
>> Depending as well if users browse a part of the application.
>>
>> We use solrj to make the queries so we did not use Readers objects
>> directly.
>>
>> Without sorting we don't see the problem
>> Regards,
>> Matteo
>>
>> On 22 June 2010 17:01, Erick Erickson <er...@gmail.com> wrote:
>> > Hmmmm.. A couple of details I'm wondering about. How many
>> > documents are we talking about in your index? Do you get
>> > OOMs when you start fresh or does it take a while?
>> >
>> > You've done some good investigations, so it seems like there
>> > could well be something else going on here than just "the usual
>> > suspects" of sorting....
>> >
>> > I'm wondering if you aren't really closing readers somehow.
>> > Are you updating your index frequently and re-opening readers often?
>> > If so, how?
>> >
>> > I'm assuming that if you do NOT sort on all these fields, you don't have
>> > the problem, is that true?
>> >
>> > Best
>> > Erick
>> >
>> > On Fri, Jun 18, 2010 at 10:52 AM, Matteo Fiandesio <
>> > matteo.fiandesio@gmail.com> wrote:
>> >
>> >> Hello,
>> >> we are experiencing OOM exceptions in our single core solr instance
>> >> (on a (huge) amazon EC2 machine).
>> >> We investigated a lot in the mailing list and through jmap/jhat dump
>> >> analyzing and the problem resides in the lucene FieldCache that fills
>> >> the heap and blows up the server.
>> >>
>> >> Our index is quite small but we have a lot of sort queries on fields
>> >> that are dynamic,of type long representing timestamps and are not
>> >> present in all the documents.
>> >> Those queries apply sorting on 12-15 of those fields.
>> >>
>> >> We are using solr 1.4 in production and the dump shows a lot of
>> >> Integer/Character and Byte Array filled up with 0s.
>> >> With solr's trunk code things does not change.
>> >>
>> >> In the mailing list we saw a lot of messages related to this issues:
>> >> we tried truncating the dates to day precision,using missingSortLast =
>> >> true,changing the field type from slong to long,setting autowarming to
>> >> different values,disabling and enabling caches with different values
>> >> but we did not manage to solve the problem.
>> >>
>> >> We were thinking to implement an LRUFieldCache field type to manage
>> >> the FieldCache as an LRU and preventing but, before starting a new
>> >> development, we want to be sure that we are not doing anything wrong
>> >> in the solr configuration or in the index generation.
>> >>
>> >> Any help would be appreciated.
>> >> Regards,
>> >> Matteo
>> >>
>> >
>>
>
Re: OOM on sorting on dynamic fields
Posted by Erick Erickson <er...@gmail.com>.
Hmmm, I'm missing something here then. Sorting over 15 fields of type long
shouldn't use much memory, even if all the values are unique. When you say
"12-15 dynamic fields", are you talking about 12-15 fields per query out of
XXX total fields? And is XXX large? At a guess, how many different fields
do
you think you're sorting over cumulative by the time you get your OOM?
Note if you sort over the field "erick_time" in 10 different queries, I'm
only counting that as 1 field. I guess another way of asking this is
"how many dynamic fields are there total?".
If this is really a sorting issue, you should be able to force this to
happen
almost immediately by firing off enough sort queries at the server. It'll
tell you a lot if you can't make this happen, even on a relatively small
test machine.
Best
Erick
On Tue, Jun 22, 2010 at 12:59 PM, Matteo Fiandesio <
matteo.fiandesio@gmail.com> wrote:
> Hi Erick,
> the index is quite small (1691145 docs) but sorting is massive and
> often on unique timestamp fields.
>
> OOM occur after a range of time between three and four hours.
> Depending as well if users browse a part of the application.
>
> We use solrj to make the queries so we did not use Readers objects
> directly.
>
> Without sorting we don't see the problem
> Regards,
> Matteo
>
> On 22 June 2010 17:01, Erick Erickson <er...@gmail.com> wrote:
> > Hmmmm.. A couple of details I'm wondering about. How many
> > documents are we talking about in your index? Do you get
> > OOMs when you start fresh or does it take a while?
> >
> > You've done some good investigations, so it seems like there
> > could well be something else going on here than just "the usual
> > suspects" of sorting....
> >
> > I'm wondering if you aren't really closing readers somehow.
> > Are you updating your index frequently and re-opening readers often?
> > If so, how?
> >
> > I'm assuming that if you do NOT sort on all these fields, you don't have
> > the problem, is that true?
> >
> > Best
> > Erick
> >
> > On Fri, Jun 18, 2010 at 10:52 AM, Matteo Fiandesio <
> > matteo.fiandesio@gmail.com> wrote:
> >
> >> Hello,
> >> we are experiencing OOM exceptions in our single core solr instance
> >> (on a (huge) amazon EC2 machine).
> >> We investigated a lot in the mailing list and through jmap/jhat dump
> >> analyzing and the problem resides in the lucene FieldCache that fills
> >> the heap and blows up the server.
> >>
> >> Our index is quite small but we have a lot of sort queries on fields
> >> that are dynamic,of type long representing timestamps and are not
> >> present in all the documents.
> >> Those queries apply sorting on 12-15 of those fields.
> >>
> >> We are using solr 1.4 in production and the dump shows a lot of
> >> Integer/Character and Byte Array filled up with 0s.
> >> With solr's trunk code things does not change.
> >>
> >> In the mailing list we saw a lot of messages related to this issues:
> >> we tried truncating the dates to day precision,using missingSortLast =
> >> true,changing the field type from slong to long,setting autowarming to
> >> different values,disabling and enabling caches with different values
> >> but we did not manage to solve the problem.
> >>
> >> We were thinking to implement an LRUFieldCache field type to manage
> >> the FieldCache as an LRU and preventing but, before starting a new
> >> development, we want to be sure that we are not doing anything wrong
> >> in the solr configuration or in the index generation.
> >>
> >> Any help would be appreciated.
> >> Regards,
> >> Matteo
> >>
> >
>
Re: OOM on sorting on dynamic fields
Posted by Matteo Fiandesio <ma...@gmail.com>.
Hi Erick,
the index is quite small (1691145 docs) but sorting is massive and
often on unique timestamp fields.
OOM occur after a range of time between three and four hours.
Depending as well if users browse a part of the application.
We use solrj to make the queries so we did not use Readers objects directly.
Without sorting we don't see the problem
Regards,
Matteo
On 22 June 2010 17:01, Erick Erickson <er...@gmail.com> wrote:
> Hmmmm.. A couple of details I'm wondering about. How many
> documents are we talking about in your index? Do you get
> OOMs when you start fresh or does it take a while?
>
> You've done some good investigations, so it seems like there
> could well be something else going on here than just "the usual
> suspects" of sorting....
>
> I'm wondering if you aren't really closing readers somehow.
> Are you updating your index frequently and re-opening readers often?
> If so, how?
>
> I'm assuming that if you do NOT sort on all these fields, you don't have
> the problem, is that true?
>
> Best
> Erick
>
> On Fri, Jun 18, 2010 at 10:52 AM, Matteo Fiandesio <
> matteo.fiandesio@gmail.com> wrote:
>
>> Hello,
>> we are experiencing OOM exceptions in our single core solr instance
>> (on a (huge) amazon EC2 machine).
>> We investigated a lot in the mailing list and through jmap/jhat dump
>> analyzing and the problem resides in the lucene FieldCache that fills
>> the heap and blows up the server.
>>
>> Our index is quite small but we have a lot of sort queries on fields
>> that are dynamic,of type long representing timestamps and are not
>> present in all the documents.
>> Those queries apply sorting on 12-15 of those fields.
>>
>> We are using solr 1.4 in production and the dump shows a lot of
>> Integer/Character and Byte Array filled up with 0s.
>> With solr's trunk code things does not change.
>>
>> In the mailing list we saw a lot of messages related to this issues:
>> we tried truncating the dates to day precision,using missingSortLast =
>> true,changing the field type from slong to long,setting autowarming to
>> different values,disabling and enabling caches with different values
>> but we did not manage to solve the problem.
>>
>> We were thinking to implement an LRUFieldCache field type to manage
>> the FieldCache as an LRU and preventing but, before starting a new
>> development, we want to be sure that we are not doing anything wrong
>> in the solr configuration or in the index generation.
>>
>> Any help would be appreciated.
>> Regards,
>> Matteo
>>
>
Re: OOM on sorting on dynamic fields
Posted by Erick Erickson <er...@gmail.com>.
Hmmmm.. A couple of details I'm wondering about. How many
documents are we talking about in your index? Do you get
OOMs when you start fresh or does it take a while?
You've done some good investigations, so it seems like there
could well be something else going on here than just "the usual
suspects" of sorting....
I'm wondering if you aren't really closing readers somehow.
Are you updating your index frequently and re-opening readers often?
If so, how?
I'm assuming that if you do NOT sort on all these fields, you don't have
the problem, is that true?
Best
Erick
On Fri, Jun 18, 2010 at 10:52 AM, Matteo Fiandesio <
matteo.fiandesio@gmail.com> wrote:
> Hello,
> we are experiencing OOM exceptions in our single core solr instance
> (on a (huge) amazon EC2 machine).
> We investigated a lot in the mailing list and through jmap/jhat dump
> analyzing and the problem resides in the lucene FieldCache that fills
> the heap and blows up the server.
>
> Our index is quite small but we have a lot of sort queries on fields
> that are dynamic,of type long representing timestamps and are not
> present in all the documents.
> Those queries apply sorting on 12-15 of those fields.
>
> We are using solr 1.4 in production and the dump shows a lot of
> Integer/Character and Byte Array filled up with 0s.
> With solr's trunk code things does not change.
>
> In the mailing list we saw a lot of messages related to this issues:
> we tried truncating the dates to day precision,using missingSortLast =
> true,changing the field type from slong to long,setting autowarming to
> different values,disabling and enabling caches with different values
> but we did not manage to solve the problem.
>
> We were thinking to implement an LRUFieldCache field type to manage
> the FieldCache as an LRU and preventing but, before starting a new
> development, we want to be sure that we are not doing anything wrong
> in the solr configuration or in the index generation.
>
> Any help would be appreciated.
> Regards,
> Matteo
>