You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Naresh Yadav <ny...@gmail.com> on 2015/09/08 15:57:11 UTC
Help on Out of memory when using Cursor with sort on Unique Key
Cluster details :
Solr Version : solr-4.10.4
No of nodes : 2 each 16 GB RAM
Node of shards : 2
Replication : 1
Each node memory parameter : -Xms2g, -Xmx4g
Collection details :
No of docs in my collection : 12.31 million
Indexed field per document : 2
Unique key field : tids
Stored filed per document : varies 30- 40
Total index size node1+node2 = 13gb+13gb=26gb
Query throwing Heap Space : /select?q=*:*&sort=tids+desc&rows=100&fl=tids
Query working* : */select?q=*:*&rows=100&fl=tids
I am using sort on unique key field tids for Cursor based pagination of 100
size.
Already tried :
I also tried tweaking Xmx but problem not solved..
I also tried q with criteria of indexed filed with only 4200 hits that also
not working
when sort parameter included.
Please help me here as i am clueless why OOM error in getting 100 documents.
Thanks
Naresh
Re: Help on Out of memory when using Cursor with sort on Unique Key
Posted by Naresh Yadav <ny...@gmail.com>.
docvalues with reindexing does not seem viable option for me as of
now...regarding second question on Xmx4G so i tried
various options Xmx8G, Xmx10G, Xmx12G all not worked except Xmx14G which
not seem practical for production with 16gb ram.
While searching i came across :
https://issues.apache.org/jira/browse/SOLR-6121 &
https://issues.apache.org/jira/browse/SOLR-6277
I think i am also in same boat where for Cursor use i have to specify
unique key field in SORT and that leads to heap space.
Please suggest some workaround for now as SOLR-6121, SOLR-627 are still in
discussion.
On Tue, Sep 8, 2015 at 7:42 PM, Raja Pothuganti <
RPothuganti@competitrack.com> wrote:
> Hi Naresh
>
> 1) For 'sort by' fields, have you considered using DocValue=true for in
> schema definition.
> If you are changing schema definition, you would need redo full reindex
> after backing up & deleting current index from dataDir.
> Also note that, adding docValue=true would increase size of index.
>
> 2)>Each node memory parameter : -Xms2g, -Xmx4g
> What is the basis choosing above memory sizes? Have you observed through
> jconsole or visual vm?
>
> Raja
> On 9/8/15, 8:57 AM, "Naresh Yadav" <ny...@gmail.com> wrote:
>
> >Cluster details :
> >
> >Solr Version : solr-4.10.4
> >No of nodes : 2 each 16 GB RAM
> >Node of shards : 2
> >Replication : 1
> >Each node memory parameter : -Xms2g, -Xmx4g
> >
> >Collection details :
> >
> >No of docs in my collection : 12.31 million
> >Indexed field per document : 2
> >Unique key field : tids
> >Stored filed per document : varies 30- 40
> >Total index size node1+node2 = 13gb+13gb=26gb
> >
> >Query throwing Heap Space : /select?q=*:*&sort=tids+desc&rows=100&fl=tids
> >
> >Query working* : */select?q=*:*&rows=100&fl=tids
> >
> >I am using sort on unique key field tids for Cursor based pagination of
> >100
> >size.
> >
> >Already tried :
> >
> >I also tried tweaking Xmx but problem not solved..
> >I also tried q with criteria of indexed filed with only 4200 hits that
> >also
> >not working
> >when sort parameter included.
> >
> >Please help me here as i am clueless why OOM error in getting 100
> >documents.
> >
> >Thanks
> >Naresh
>
>
Re: Help on Out of memory when using Cursor with sort on Unique Key
Posted by Raja Pothuganti <RP...@competitrack.com>.
Hi Naresh
1) For 'sort by' fields, have you considered using DocValue=true for in
schema definition.
If you are changing schema definition, you would need redo full reindex
after backing up & deleting current index from dataDir.
Also note that, adding docValue=true would increase size of index.
2)>Each node memory parameter : -Xms2g, -Xmx4g
What is the basis choosing above memory sizes? Have you observed through
jconsole or visual vm?
Raja
On 9/8/15, 8:57 AM, "Naresh Yadav" <ny...@gmail.com> wrote:
>Cluster details :
>
>Solr Version : solr-4.10.4
>No of nodes : 2 each 16 GB RAM
>Node of shards : 2
>Replication : 1
>Each node memory parameter : -Xms2g, -Xmx4g
>
>Collection details :
>
>No of docs in my collection : 12.31 million
>Indexed field per document : 2
>Unique key field : tids
>Stored filed per document : varies 30- 40
>Total index size node1+node2 = 13gb+13gb=26gb
>
>Query throwing Heap Space : /select?q=*:*&sort=tids+desc&rows=100&fl=tids
>
>Query working* : */select?q=*:*&rows=100&fl=tids
>
>I am using sort on unique key field tids for Cursor based pagination of
>100
>size.
>
>Already tried :
>
>I also tried tweaking Xmx but problem not solved..
>I also tried q with criteria of indexed filed with only 4200 hits that
>also
>not working
>when sort parameter included.
>
>Please help me here as i am clueless why OOM error in getting 100
>documents.
>
>Thanks
>Naresh