You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Krzysztof Szlapinski <kr...@starline.hk> on 2008/11/19 15:45:24 UTC
crashes at a heavy load

hi
im running some test of hbase on hadoop
when namenode is heavily overloaded it crushes with the following error

java.io.IOException: Connection reset by peer                                                                                                                
        at sun.nio.ch.FileDispatcher.write0(Native Method)                                                                                                   
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)                                                                                       
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)                                                                                          
        at sun.nio.ch.IOUtil.write(IOUtil.java:75)                                                                                                           
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)                                                                                    
        at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)                                                             
        at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:140)                                                                      
        at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)                                                                       
        at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)                                                                       
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)                                                                            
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)                                                                                 
        at java.io.DataOutputStream.flush(DataOutputStream.java:106)                                                                                         
        at org.apache.hadoop.dfs.DataNode$BlockSender.sendBlock(DataNode.java:2019)                                                                          
        at org.apache.hadoop.dfs.DataNode$DataXceiver.readBlock(DataNode.java:1140)                                                                          
        at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:1068)                                                                                
        at java.lang.Thread.run(Thread.java:619)                                                                                                             
                                                                                                                                                             
2008-11-19 14:52:42,007 ERROR org.apache.hadoop.dfs.DataNode: DatanodeRegistration(192.168.1.109:50010, storageID=DS-1340454997-192.168.1.109-50010-122709122
1665, infoPort=50075, ipcPort=50020):DataXceiver: java.io.IOException: Connection reset by peer                                                              
        at sun.nio.ch.FileDispatcher.write0(Native Method)                                                                                                   
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)                                                                                       
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)                                                                                          
        at sun.nio.ch.IOUtil.write(IOUtil.java:75)                                                                                                           
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)                                                                                    
        at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)                                                             
        at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:140)                                                                      
        at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)                                                                       
        at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)                                                                       
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)                                                                            
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)                                                                                 
        at java.io.DataOutputStream.flush(DataOutputStream.java:106)                                                                                         
        at org.apache.hadoop.dfs.DataNode$BlockSender.sendBlock(DataNode.java:2019)                                                                          
        at org.apache.hadoop.dfs.DataNode$DataXceiver.readBlock(DataNode.java:1140)                                                                          
        at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:1068)                                                                                
        at java.lang.Thread.run(Thread.java:619)


and throws Could not obtain blk as a result
Can I tune hadoop in some way to prevent this error?
Im using the trunk version of hadoop

krzysiek