Canopy generation out of memory troubleshooting

[Chih-Hsien Wu <ch...@gmail.com>]

Hi All,  I posted up a Mahout canopy generation related troubleshoot
last week; however, I didn't get the problem solved. The message below
is the error I received. I'm trying to run canopy generation about 900
mb worth of information. There are estimated about 120,000 vectors.
I'm currently running this on Amazon EMR  two m1.xlarge instances.

I noticed that memory is not used fully on the hadoop machines right
before error occurred. I looked into tweaking namenode, datanode heap
sizes, and jvm for each task; but I still can't optimize the memory
usage on the machines. I'm currently setting "mapred.child.java.opts"
to 8gb while datanode_heap_size to 3g. I'm looking for explanation on
memory setting to solve this problem.


2013-11-27 16:15:01,6423 ERROR Client
fs/client/fileclient/cc/client.cc:3769 Thread: 140066469238528 rpc err
Connection timed out(110) 28.9 to 10.60.9.161:5660, fid
2051.559.132980, upd 0
2013-11-27 16:19:35,9552 ERROR Client
fs/client/fileclient/cc/client.cc:3769 Thread: 140066469238528 rpc err
Connection reset by peer(104) 28.122 to 10.167.5.31:5660, fid
2168.36.131182, upd 0
Error: Java heap space
Error: GC overhead limit exceeded
Error: GC overhead limit exceeded
attempt_201311271453_0362_m_000000_0: Exception in thread
"communication thread" java.lang.OutOfMemoryError: GC overhead limit
exceeded
attempt_201311271453_0362_m_000000_0: 	at
java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:64)
attempt_201311271453_0362_m_000000_0: 	at
java.lang.StringBuilder.<init>(StringBuilder.java:85)
Error: Java heap space
Error: Java heap space
Error: Java heap space
java.lang.InterruptedException: Canopy Job failed processing
maprfs:/user/hadoop/bottom/data/24
	at org.apache.mahout.clustering.canopy.CanopyDriver.buildClustersMR(CanopyDriver.java:361)
	at org.apache.mahout.clustering.canopy.CanopyDriver.buildClusters(CanopyDriver.java:246)
	at org.apache.mahout.clustering.canopy.CanopyDriver.run(CanopyDriver.java:154)
	at org.apache.mahout.clustering.canopy.CanopyDriver.run(CanopyDriver.java:168)
	at clustering.TopDownClusteringDriver.TopDownClustering(TopDownClusteringDriver.java:78)
	at clustering.TopDownClusteringDriver.main(TopDownClusteringDriver.java:133)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:197)