You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Ajinkya Kale <ka...@gmail.com> on 2015/09/28 20:46:21 UTC
Solr java.lang.OutOfMemoryError: Java heap space
Hi,
I am trying to retrieve all the documents from a solr index in a batched
manner.
I have 100M documents. I am retrieving them using the method proposed here
https://nowontap.wordpress.com/2014/04/04/solr-exporting-an-index-to-an-external-file/
I am dumping 10M document splits in each file. I get "OutOfMemoryError" if
start is at 50M. I get the same error even if rows=10 for start=50M.
Curl on start=0 rows=50M in one go works fine too. But things go bad when
start is at 50M.
My Solr version is 4.4.0.
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.lucene.search.TopDocsCollector.topDocs(TopDocsCollector.java:146)
at
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1502)
at
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1363)
at
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:474)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:434)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1904)
--aj
Re: Solr java.lang.OutOfMemoryError: Java heap space
Posted by Gili Nachum <gi...@gmail.com>.
If you can't use CursorMark, then I suggest not using the start parameter,
instead sort asc by a unique field and and range the query to records with
a field value larger then the last doc you read. Then set rows to be
whatever you found can fit in memory.
On Mon, Sep 28, 2015 at 10:59 PM, Ajinkya Kale <ka...@gmail.com>
wrote:
> If I am not wrong this works only with Solr version > 4.7.0 ?
> On Mon, Sep 28, 2015 at 12:23 PM Markus Jelsma <markus.jelsma@openindex.io
> >
> wrote:
>
> > Hi - you need to use the CursorMark feature for larger sets:
> > https://cwiki.apache.org/confluence/display/solr/Pagination+of+Results
> > M.
> >
> >
> >
> > -----Original message-----
> > > From:Ajinkya Kale <ka...@gmail.com>
> > > Sent: Monday 28th September 2015 20:46
> > > To: solr-user@lucene.apache.org; java-user@lucene.apache.org
> > > Subject: Solr java.lang.OutOfMemoryError: Java heap space
> > >
> > > Hi,
> > >
> > > I am trying to retrieve all the documents from a solr index in a
> batched
> > > manner.
> > > I have 100M documents. I am retrieving them using the method proposed
> > here
> > >
> >
> https://nowontap.wordpress.com/2014/04/04/solr-exporting-an-index-to-an-external-file/
> > > I am dumping 10M document splits in each file. I get "OutOfMemoryError"
> > if
> > > start is at 50M. I get the same error even if rows=10 for start=50M.
> > > Curl on start=0 rows=50M in one go works fine too. But things go bad
> when
> > > start is at 50M.
> > > My Solr version is 4.4.0.
> > >
> > > Caused by: java.lang.OutOfMemoryError: Java heap space
> > > at
> > >
> >
> org.apache.lucene.search.TopDocsCollector.topDocs(TopDocsCollector.java:146)
> > > at
> > >
> >
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1502)
> > > at
> > >
> >
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1363)
> > > at
> > >
> >
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:474)
> > > at
> > >
> >
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:434)
> > > at
> > >
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208)
> > > at
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1904)
> > >
> > > --aj
> > >
> >
>
Re: Solr java.lang.OutOfMemoryError: Java heap space
Posted by Ajinkya Kale <ka...@gmail.com>.
If I am not wrong this works only with Solr version > 4.7.0 ?
On Mon, Sep 28, 2015 at 12:23 PM Markus Jelsma <ma...@openindex.io>
wrote:
> Hi - you need to use the CursorMark feature for larger sets:
> https://cwiki.apache.org/confluence/display/solr/Pagination+of+Results
> M.
>
>
>
> -----Original message-----
> > From:Ajinkya Kale <ka...@gmail.com>
> > Sent: Monday 28th September 2015 20:46
> > To: solr-user@lucene.apache.org; java-user@lucene.apache.org
> > Subject: Solr java.lang.OutOfMemoryError: Java heap space
> >
> > Hi,
> >
> > I am trying to retrieve all the documents from a solr index in a batched
> > manner.
> > I have 100M documents. I am retrieving them using the method proposed
> here
> >
> https://nowontap.wordpress.com/2014/04/04/solr-exporting-an-index-to-an-external-file/
> > I am dumping 10M document splits in each file. I get "OutOfMemoryError"
> if
> > start is at 50M. I get the same error even if rows=10 for start=50M.
> > Curl on start=0 rows=50M in one go works fine too. But things go bad when
> > start is at 50M.
> > My Solr version is 4.4.0.
> >
> > Caused by: java.lang.OutOfMemoryError: Java heap space
> > at
> >
> org.apache.lucene.search.TopDocsCollector.topDocs(TopDocsCollector.java:146)
> > at
> >
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1502)
> > at
> >
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1363)
> > at
> >
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:474)
> > at
> >
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:434)
> > at
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208)
> > at
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1904)
> >
> > --aj
> >
>
RE: Solr java.lang.OutOfMemoryError: Java heap space
Posted by Markus Jelsma <ma...@openindex.io>.
Hi - you need to use the CursorMark feature for larger sets: https://cwiki.apache.org/confluence/display/solr/Pagination+of+Results
M.
-----Original message-----
> From:Ajinkya Kale <ka...@gmail.com>
> Sent: Monday 28th September 2015 20:46
> To: solr-user@lucene.apache.org; java-user@lucene.apache.org
> Subject: Solr java.lang.OutOfMemoryError: Java heap space
>
> Hi,
>
> I am trying to retrieve all the documents from a solr index in a batched
> manner.
> I have 100M documents. I am retrieving them using the method proposed here
> https://nowontap.wordpress.com/2014/04/04/solr-exporting-an-index-to-an-external-file/
> I am dumping 10M document splits in each file. I get "OutOfMemoryError" if
> start is at 50M. I get the same error even if rows=10 for start=50M.
> Curl on start=0 rows=50M in one go works fine too. But things go bad when
> start is at 50M.
> My Solr version is 4.4.0.
>
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.lucene.search.TopDocsCollector.topDocs(TopDocsCollector.java:146)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1502)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1363)
> at
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:474)
> at
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:434)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1904)
>
> --aj
>
RE: Solr java.lang.OutOfMemoryError: Java heap space
Posted by will martin <wm...@gmail.com>.
http://opensourceconnections.com/blog/2014/07/13/reindexing-collections-with-solrs-cursor-support/
-----Original Message-----
From: Ajinkya Kale [mailto:kaleajinkya@gmail.com]
Sent: Monday, September 28, 2015 2:46 PM
To: solr-user@lucene.apache.org; java-user@lucene.apache.org
Subject: Solr java.lang.OutOfMemoryError: Java heap space
Hi,
I am trying to retrieve all the documents from a solr index in a batched manner.
I have 100M documents. I am retrieving them using the method proposed here https://nowontap.wordpress.com/2014/04/04/solr-exporting-an-index-to-an-external-file/
I am dumping 10M document splits in each file. I get "OutOfMemoryError" if start is at 50M. I get the same error even if rows=10 for start=50M.
Curl on start=0 rows=50M in one go works fine too. But things go bad when start is at 50M.
My Solr version is 4.4.0.
Caused by: java.lang.OutOfMemoryError: Java heap space at
org.apache.lucene.search.TopDocsCollector.topDocs(TopDocsCollector.java:146)
at
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1502)
at
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1363)
at
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:474)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:434)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1904)
--aj
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Solr java.lang.OutOfMemoryError: Java heap space
Posted by will martin <wm...@gmail.com>.
http://opensourceconnections.com/blog/2014/07/13/reindexing-collections-with-solrs-cursor-support/
-----Original Message-----
From: Ajinkya Kale [mailto:kaleajinkya@gmail.com]
Sent: Monday, September 28, 2015 2:46 PM
To: solr-user@lucene.apache.org; java-user@lucene.apache.org
Subject: Solr java.lang.OutOfMemoryError: Java heap space
Hi,
I am trying to retrieve all the documents from a solr index in a batched manner.
I have 100M documents. I am retrieving them using the method proposed here https://nowontap.wordpress.com/2014/04/04/solr-exporting-an-index-to-an-external-file/
I am dumping 10M document splits in each file. I get "OutOfMemoryError" if start is at 50M. I get the same error even if rows=10 for start=50M.
Curl on start=0 rows=50M in one go works fine too. But things go bad when start is at 50M.
My Solr version is 4.4.0.
Caused by: java.lang.OutOfMemoryError: Java heap space at
org.apache.lucene.search.TopDocsCollector.topDocs(TopDocsCollector.java:146)
at
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1502)
at
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1363)
at
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:474)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:434)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1904)
--aj