You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by sivaprasad <si...@echidnainc.com> on 2010/12/07 06:03:55 UTC
Out of memory error
Hi,
When i am trying to import the data using DIH, iam getting Out of memory
error.The below are the configurations which i have.
Database:Mysql
Os:windows
No Of documents:15525532
In Db-config.xml i made batch size as "-1"
The solr server is running on Linux machine with tomcat.
i set tomcat arguments as ./startup.sh -Xms1024M -Xmx2048M
Can anybody has idea, where the things are going wrong?
Regards,
JS
--
View this message in context: http://lucene.472066.n3.nabble.com/Out-of-memory-error-tp2031761p2031761.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Out of memory error
Posted by Fuad Efendi <fu...@efendi.ca>.
Related: SOLR-846
Sent on the TELUS Mobility network with BlackBerry
-----Original Message-----
From: Erick Erickson <er...@gmail.com>
Date: Tue, 7 Dec 2010 08:11:41
To: <so...@lucene.apache.org>
Reply-To: solr-user@lucene.apache.org
Subject: Re: Out of memory error
Have you seen this page? http://wiki.apache.org/solr/DataImportHandlerFaq
<http://wiki.apache.org/solr/DataImportHandlerFaq>See especially batchsize,
but it looks like you're already on to that.
Do you have any idea how big the records are in the database? You might
try adjusting the rambuffersize down, what is it at now?
In general, what are our Solr commit options?
Does anything get to Solr or is the OOM when the SQL is executed?
The first question to answer is whether you index anything at all...
There's a little-know DIH debug page you can access at:
.../solr/admin/dataimport.jsp that might help, and progress can be monitored
at:
.../solr/dataimport
DIH can be "interesting", you get finer control with SolrJ and a direct
JDBC connection. If you don't get anywhere with DIH.
Scattergun response, but things to try...
Best
Erick
On Tue, Dec 7, 2010 at 12:03 AM, sivaprasad <si...@echidnainc.com>wrote:
>
> Hi,
>
> When i am trying to import the data using DIH, iam getting Out of memory
> error.The below are the configurations which i have.
>
> Database:Mysql
> Os:windows
> No Of documents:15525532
> In Db-config.xml i made batch size as "-1"
>
> The solr server is running on Linux machine with tomcat.
> i set tomcat arguments as ./startup.sh -Xms1024M -Xmx2048M
>
> Can anybody has idea, where the things are going wrong?
>
> Regards,
> JS
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Out-of-memory-error-tp2031761p2031761.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Out of memory error
Posted by Erick Erickson <er...@gmail.com>.
Have you seen this page? http://wiki.apache.org/solr/DataImportHandlerFaq
<http://wiki.apache.org/solr/DataImportHandlerFaq>See especially batchsize,
but it looks like you're already on to that.
Do you have any idea how big the records are in the database? You might
try adjusting the rambuffersize down, what is it at now?
In general, what are our Solr commit options?
Does anything get to Solr or is the OOM when the SQL is executed?
The first question to answer is whether you index anything at all...
There's a little-know DIH debug page you can access at:
.../solr/admin/dataimport.jsp that might help, and progress can be monitored
at:
.../solr/dataimport
DIH can be "interesting", you get finer control with SolrJ and a direct
JDBC connection. If you don't get anywhere with DIH.
Scattergun response, but things to try...
Best
Erick
On Tue, Dec 7, 2010 at 12:03 AM, sivaprasad <si...@echidnainc.com>wrote:
>
> Hi,
>
> When i am trying to import the data using DIH, iam getting Out of memory
> error.The below are the configurations which i have.
>
> Database:Mysql
> Os:windows
> No Of documents:15525532
> In Db-config.xml i made batch size as "-1"
>
> The solr server is running on Linux machine with tomcat.
> i set tomcat arguments as ./startup.sh -Xms1024M -Xmx2048M
>
> Can anybody has idea, where the things are going wrong?
>
> Regards,
> JS
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Out-of-memory-error-tp2031761p2031761.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>