Nutch + Solr - Indexer causes java.lang.OutOfMemoryError: Java heap space

[glumet <ja...@gmail.com>]

Hello everyone, 

I have configured my 2 servers to run in distributed mode (with Hadoop) and
my configuration for crawling process is Nutch 2.2.1 - HBase (as a storage)
and Solr. Solr is run by Tomcat. The problem is everytime I try to do the
last step - I mean when I want to index data from HBase into Solr. After
then this *[1]* error occures. I tried to add CATALINA_OPTS (or JAVA_OPTS)
like this:

CATALINA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -Xms1g -Xmx6000m
-XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=30 -XX:MaxPermSize=512m
-XX:+CMSClassUnloadingEnabled"

to Tomcat's catalina.sh script and run server with this script but it didn't
help. I also add these *[2]* properties to nutch-site.xml file but it ended
up with OutOfMemory again. Can you help me please?

*[1]*
/2014-09-06 22:52:50,683 FATAL org.apache.hadoop.mapred.Child: Error running
child : java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:2367)
	at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
	at
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:587)
	at java.lang.StringBuffer.append(StringBuffer.java:332)
	at java.io.StringWriter.write(StringWriter.java:77)
	at org.apache.solr.common.util.XML.escape(XML.java:204)
	at org.apache.solr.common.util.XML.escapeCharData(XML.java:77)
	at org.apache.solr.common.util.XML.writeXML(XML.java:147)
	at
org.apache.solr.client.solrj.util.ClientUtils.writeVal(ClientUtils.java:161)
	at
org.apache.solr.client.solrj.util.ClientUtils.writeXML(ClientUtils.java:129)
	at
org.apache.solr.client.solrj.request.UpdateRequest.writeXML(UpdateRequest.java:355)
	at
org.apache.solr.client.solrj.request.UpdateRequest.getXML(UpdateRequest.java:271)
	at
org.apache.solr.client.solrj.request.RequestWriter.getContentStream(RequestWriter.java:66)
	at
org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getDelegate(RequestWriter.java:94)
	at
org.apache.solr.client.solrj.request.RequestWriter$LazyContentStream.getName(RequestWriter.java:104)
	at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:247)
	at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:197)
	at
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:68)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54)
	at
org.apache.nutch.indexwriter.solr.SolrIndexWriter.close(SolrIndexWriter.java:96)
	at org.apache.nutch.indexer.IndexWriters.close(IndexWriters.java:117)
	at
org.apache.nutch.indexer.IndexerOutputFormat$1.close(IndexerOutputFormat.java:54)
	at
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:650)
	at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1793)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:779)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
/

*[2]*

<property>
  <name>http.content.limit</name>
  <value>150000000</value>
  <description>The length limit for downloaded content using the http
  protocol, in bytes. If this value is nonnegative (>=0), content longer
  than it will be truncated; otherwise, no truncation at all. Do not
  confuse this setting with the file.content.limit setting.
  For our purposes it is twice bigger than default - parsing big pages: 128
* 1024
  </description>
</property>

<property>
   <name>indexer.max.tokens</name>
   <value>100000</value>
</property>

<property>
  <name>http.timeout</name>
  <value>50000</value>
  <description>The default network timeout, in milliseconds.</description>
</property>

<property>
  <name>solr.commit.size</name>
  <value>100</value>
  <description>
  Defines the number of documents to send to Solr in a single update batch.
  Decrease when handling very large documents to prevent Nutch from running
  out of memory. NOTE: It does not explicitly trigger a server side commit.
  </description>
</property>



--
View this message in context: http://lucene.472066.n3.nabble.com/Nutch-Solr-Indexer-causes-java-lang-OutOfMemoryError-Java-heap-space-tp4157308.html
Sent from the Solr - User mailing list archive at Nabble.com.