You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Harsh J <ha...@cloudera.com> on 2012/02/01 04:23:27 UTC

Re: Exception in doCheckpoint

Gabriel,

Do you get a download if you visit http://master:50070/getimage in
your browser or do you get a 404? It is likely that the URL threw a
404 which lead to the FileNotFoundException. Possibly something wrong
with the NN webapp server.

On Wed, Feb 1, 2012 at 3:19 AM, Gabriel Rosendorf
<gr...@e3smartenergy.com> wrote:
> I'm getting this recurring exception on my secondary NameNode.
>
> 2012-01-31 21:46:12,710 INFO org.apache.hadoop.hdfs.util.GSet: recommended=2097152, actual=2097152
> 2012-01-31 21:46:12,712 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=hduser
> 2012-01-31 21:46:12,713 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup
> 2012-01-31 21:46:12,713 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true
> 2012-01-31 21:46:12,713 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.block.invalidate.limit=100
> 2012-01-31 21:46:12,713 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
> 2012-01-31 21:46:12,713 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
> 2012-01-31 21:46:12,714 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 6
> 2012-01-31 21:46:12,715 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files under construction = 0
> 2012-01-31 21:46:12,715 INFO org.apache.hadoop.hdfs.server.common.Storage: Edits file /app/hadoop/dfs/namesecondary/current/edits of size 4 edits # 0 loaded in 0 seconds.
> 2012-01-31 21:46:12,715 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 0 Total time for transactions(ms): 0Number of transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0
> 2012-01-31 21:46:12,720 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 559 saved in 0 seconds.
> 2012-01-31 21:46:12,735 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 559 saved in 0 seconds.
> 2012-01-31 21:46:12,869 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Posted URL master:50070putimage=1&port=50090&machine=0.0.0.0&token=-31:2049933803:0:1328044870000:1328044562914
> 2012-01-31 21:46:12,876 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint:
> 2012-01-31 21:46:12,876 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.io.FileNotFoundException: http://master:50070/getimage?putimage=1&port=50090&machine=0.0.0.0&token=-31:2049933803:0:1328044870000:1328044562914
>        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1434)
>        at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
>        at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.putFSImage(SecondaryNameNode.java:377)
>        at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:418)
>        at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
>        at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
>        at java.lang.Thread.run(Thread.java:662)
>
> Any ideas?
>
> Best,
> Gabriel Rosendorf



-- 
Harsh J
Customer Ops. Engineer
Cloudera | http://tiny.cloudera.com/about

Re: Exception in doCheckpoint

Posted by Gabriel Rosendorf <gr...@e3smartenergy.com>.
Harsh,

That was part of the problem.  I fixed hdfs-site.xml, and now I can connect to the http://master:50070, but when I hit http://master:50070/getimage, I get a 401.  Also, I'm now seeing this in the log on my 2NN:

2012-02-01 15:17:04,896 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Web server init done
2012-02-01 15:17:04,896 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Secondary Web-server up at: 0.0.0.0:50090
2012-02-01 15:17:04,896 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Secondary image servlet up at: 0.0.0.0:50090
2012-02-01 15:17:04,896 WARN org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Checkpoint Period   :3600 secs (60 min)
2012-02-01 15:17:04,896 WARN org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Log Size Trigger    :67108864 bytes (65536 KB)
2012-02-01 15:22:04,967 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint: 
2012-02-01 15:22:04,968 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.net.ConnectException: Connection refused
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
	at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:211)
	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
	at java.net.Socket.connect(Socket.java:529)
	at java.net.Socket.connect(Socket.java:478)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:233)
	at sun.net.www.http.HttpClient.New(HttpClient.java:306)
	at sun.net.www.http.HttpClient.New(HttpClient.java:323)
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911)
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172)
	at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
	at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
	at java.lang.Thread.run(Thread.java:662)


-Gabe

On Jan 31, 2012, at 10:23 PM, Harsh J wrote:

> Gabriel,
> 
> Do you get a download if you visit http://master:50070/getimage in
> your browser or do you get a 404? It is likely that the URL threw a
> 404 which lead to the FileNotFoundException. Possibly something wrong
> with the NN webapp server.
> 
> On Wed, Feb 1, 2012 at 3:19 AM, Gabriel Rosendorf
> <gr...@e3smartenergy.com> wrote:
>> I'm getting this recurring exception on my secondary NameNode.
>> 
>> 2012-01-31 21:46:12,710 INFO org.apache.hadoop.hdfs.util.GSet: recommended=2097152, actual=2097152
>> 2012-01-31 21:46:12,712 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=hduser
>> 2012-01-31 21:46:12,713 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup
>> 2012-01-31 21:46:12,713 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=true
>> 2012-01-31 21:46:12,713 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.block.invalidate.limit=100
>> 2012-01-31 21:46:12,713 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
>> 2012-01-31 21:46:12,713 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring more than 10 times
>> 2012-01-31 21:46:12,714 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 6
>> 2012-01-31 21:46:12,715 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files under construction = 0
>> 2012-01-31 21:46:12,715 INFO org.apache.hadoop.hdfs.server.common.Storage: Edits file /app/hadoop/dfs/namesecondary/current/edits of size 4 edits # 0 loaded in 0 seconds.
>> 2012-01-31 21:46:12,715 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 0 Total time for transactions(ms): 0Number of transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0
>> 2012-01-31 21:46:12,720 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 559 saved in 0 seconds.
>> 2012-01-31 21:46:12,735 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 559 saved in 0 seconds.
>> 2012-01-31 21:46:12,869 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Posted URL master:50070putimage=1&port=50090&machine=0.0.0.0&token=-31:2049933803:0:1328044870000:1328044562914
>> 2012-01-31 21:46:12,876 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint:
>> 2012-01-31 21:46:12,876 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: java.io.FileNotFoundException: http://master:50070/getimage?putimage=1&port=50090&machine=0.0.0.0&token=-31:2049933803:0:1328044870000:1328044562914
>>        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1434)
>>        at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
>>        at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.putFSImage(SecondaryNameNode.java:377)
>>        at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:418)
>>        at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
>>        at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
>>        at java.lang.Thread.run(Thread.java:662)
>> 
>> Any ideas?
>> 
>> Best,
>> Gabriel Rosendorf
> 
> 
> 
> -- 
> Harsh J
> Customer Ops. Engineer
> Cloudera | http://tiny.cloudera.com/about