You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Stuti Awasthi <st...@hcl.com> on 2014/01/31 14:14:48 UTC
java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<http://%3cHostname%3e:50070/dfshealth.jsp> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
........
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Posted by Jitendra Yadav <je...@gmail.com>.
Correcting typo error.
*<name>dfs.namenode.http-address</name>*
*Thanks*
On Fri, Jan 31, 2014 at 7:25 PM, Jitendra Yadav
<je...@gmail.com>wrote:
> Can you please change below property and restart your cluster again?
>
> FROM:
>
> <name>dfs.http.address</name>
>
>
> TO:
> <name>dfs.namenode.http-addres</name>
>
> Thanks
> Jitendra
>
>
> On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>wrote:
>
>> Hi Jitendra,
>>
>>
>>
>> I realized that some days back ,my cluster was down due to power failure
>> after which nn/current directory has : edits, edits.new file and now SNN is
>> not rolling these edits due to HTTP error.
>>
>> Also currently my NN and SNN are operating on same machine
>>
>>
>>
>>
>>
>> DFSadmin report :
>>
>>
>>
>> Configured Capacity: 659494076416 (614.2 GB)
>>
>> Present Capacity: 535599210496 (498.82 GB)
>>
>> DFS Remaining: 497454006272 (463.29 GB)
>>
>> DFS Used: 38145204224 (35.53 GB)
>>
>> DFS Used%: 7.12%
>>
>> Under replicated blocks: 283
>>
>> Blocks with corrupt replicas: 3
>>
>> Missing blocks: 3
>>
>>
>>
>> -------------------------------------------------
>>
>> Datanodes available: 8 (8 total, 0 dead)
>>
>>
>>
>> Name: 10.139.9.238:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 4302274560 (4.01 GB)
>>
>> Non DFS Used: 8391843840 (7.82 GB)
>>
>> DFS Remaining: 69742641152(64.95 GB)
>>
>> DFS Used%: 5.22%
>>
>> DFS Remaining%: 84.6%
>>
>> Last contact: Fri Jan 31 18:55:18 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.233:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 5774745600 (5.38 GB)
>>
>> Non DFS Used: 13409488896 (12.49 GB)
>>
>> DFS Remaining: 63252525056(58.91 GB)
>>
>> DFS Used%: 7.01%
>>
>> DFS Remaining%: 76.73%
>>
>> Last contact: Fri Jan 31 18:55:19 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.232:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 8524451840 (7.94 GB)
>>
>> Non DFS Used: 24847884288 (23.14 GB)
>>
>> DFS Remaining: 49064423424(45.69 GB)
>>
>> DFS Used%: 10.34%
>>
>> DFS Remaining%: 59.52%
>>
>> Last contact: Fri Jan 31 18:55:21 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.236:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 4543819776 (4.23 GB)
>>
>> Non DFS Used: 8669548544 (8.07 GB)
>>
>> DFS Remaining: 69223391232(64.47 GB)
>>
>> DFS Used%: 5.51%
>>
>> DFS Remaining%: 83.97%
>>
>> Last contact: Fri Jan 31 18:55:19 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.235:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 5092986880 (4.74 GB)
>>
>> Non DFS Used: 8669454336 (8.07 GB)
>>
>> DFS Remaining: 68674318336(63.96 GB)
>>
>> DFS Used%: 6.18%
>>
>> DFS Remaining%: 83.31%
>>
>> Last contact: Fri Jan 31 18:55:19 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.237:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 4604301312 (4.29 GB)
>>
>> Non DFS Used: 11005788160 (10.25 GB)
>>
>> DFS Remaining: 66826670080(62.24 GB)
>>
>> DFS Used%: 5.59%
>>
>> DFS Remaining%: 81.06%
>>
>> Last contact: Fri Jan 31 18:55:18 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.234:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 4277760000 (3.98 GB)
>>
>> Non DFS Used: 12124221440 (11.29 GB)
>>
>> DFS Remaining: 66034778112(61.5 GB)
>>
>> DFS Used%: 5.19%
>>
>> DFS Remaining%: 80.1%
>>
>> Last contact: Fri Jan 31 18:55:18 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.231:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 1024864256 (977.39 MB)
>>
>> Non DFS Used: 36776636416 (34.25 GB)
>>
>> DFS Remaining: 44635258880(41.57 GB)
>>
>> DFS Used%: 1.24%
>>
>> DFS Remaining%: 54.14%
>>
>> Last contact: Fri Jan 31 18:55:20 IST 2014
>>
>>
>>
>>
>>
>>
>>
>> *From:* Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
>> *Sent:* Friday, January 31, 2014 6:58 PM
>> *To:* user
>> *Subject:* Re: java.io.FileNotFoundException:
>> http://HOSTNAME:50070/getimage?getimage=1
>>
>>
>>
>> Hi,
>>
>>
>>
>> Please post the output of dfs report command, this could help us to
>> understand cluster health.
>>
>>
>>
>> # *hadoop dfsadmin -report*
>>
>>
>>
>> Thanks
>>
>> Jitendra
>>
>>
>>
>> On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>
>> wrote:
>>
>> Hi All,
>>
>>
>>
>> I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP
>> request at port 50070 on dfs is not working properly.
>>
>> I have an Hadoop cluster which is operating from several days. Recently
>> we are also not able to see dfshealth.jsp page from webconsole.
>>
>>
>>
>> Problems :
>>
>> 1. http://<Hostname>:50070/dfshealth.jsp shows following error
>>
>>
>>
>> HTTP ERROR: 404
>>
>> Problem accessing /. Reason:
>>
>> NOT_FOUND
>>
>>
>>
>> 2. SNN is not able to roll edits :
>>
>> ERROR in SecondaryNameNode Log
>>
>> java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
>>
>> at
>> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
>>
>> at java.security.AccessController.doPrivileged(Native Method)
>>
>> at javax.security.auth.Subject.doAs(Subject.java:416)
>>
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
>>
>>
>>
>> ERROR in Namenode Log
>>
>> 2014-01-31 18:15:12,046 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>> 10.139.9.231
>>
>> 2014-01-31 18:15:12,046 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log,
>> edits.new files already exists in all healthy directories:
>>
>> /usr/lib/hadoop/storage/dfs/nn/current/edits.new
>>
>>
>>
>>
>>
>>
>>
>> Namenode logs which suggest that webserver is started on 50070
>> successfully:
>>
>> 2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port
>> returned by webServer.getConnectors()[0].getLocalPort() before open() is
>> -1. Opening the listener on 50070
>>
>> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer:
>> listener.getLocalPort() returned 50070
>> webServer.getConnectors()[0].getLocalPort() returned 50070
>>
>> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty
>> bound to port 50070
>>
>> 2014-01-31 14:42:35,378 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
>> HOSTNAME:50070
>>
>>
>>
>>
>>
>> Hdfs-site.xml
>>
>> <configuration>
>>
>> <property>
>>
>> <name>dfs.replication</name>
>>
>> <value>2</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>dfs.name.dir</name>
>>
>> <value>/usr/lib/hadoop/storage/dfs/nn</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>dfs.data.dir</name>
>>
>> <value>/usr/lib/hadoop/storage/dfs/dn</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>dfs.permissions</name>
>>
>> <value>false</value>
>>
>> </property>
>>
>> <property>
>>
>> <name>dfs.webhdfs.enabled</name>
>>
>> <value>true</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>dfs.http.address</name>
>>
>> <value>HOSTNAME:50070</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>dfs.secondary.http.address</name>
>>
>> <value>HOSTNAME:50090</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>fs.checkpoint.dir</name>
>>
>> <value>/usr/lib/hadoop/storage/dfs/snn</value>
>>
>> </property>
>>
>>
>>
>> </configuration>
>>
>>
>>
>>
>>
>> */etc/hosts* (Note I have also tried by commenting 127.0.0.1 entry in
>> host file but the issue was not resolved)
>>
>>
>>
>> 127.0.0.1 localhost
>>
>>
>>
>> IP1 Hostname1 # Namenode- vm01 - itself
>>
>> IP2 Hostname2 # DataNode- vm02
>>
>> ........
>>
>>
>>
>> # The following lines are desirable for IPv6 capable hosts
>>
>> ::1 ip6-localhost ip6-loopback
>>
>> fe00::0 ip6-localnet
>>
>> ff00::0 ip6-mcastprefix
>>
>> ff02::1 ip6-allnodes
>>
>> ff02::2 ip6-allrouters
>>
>>
>>
>>
>>
>> Note : All Hadoop daemons are executing fine and the jobs are running
>> properly.
>>
>>
>>
>> How to resolve this issue, I have tried many options provided on
>> different forums but still facing the same issue.
>>
>> I belive that this can cause a major problem later as my edits are not
>> getting rolled into fsimage.. This can cause me a data loss in case of
>> failure.
>>
>>
>>
>> Please suggest
>>
>>
>>
>> Thanks
>>
>> Stuti
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ::DISCLAIMER::
>>
>> ----------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> The contents of this e-mail and any attachment(s) are confidential and
>> intended for the named recipient(s) only.
>> E-mail transmission is not guaranteed to be secure or error-free as
>> information could be intercepted, corrupted,
>> lost, destroyed, arrive late or incomplete, or may contain viruses in
>> transmission. The e mail and its contents
>> (with or without referred errors) shall therefore not attach any
>> liability on the originator or HCL or its affiliates.
>> Views or opinions, if any, presented in this email are solely those of
>> the author and may not necessarily reflect the
>> views or opinions of HCL or its affiliates. Any form of reproduction,
>> dissemination, copying, disclosure, modification,
>> distribution and / or publication of this message without the prior
>> written consent of authorized representative of
>> HCL is strictly prohibited. If you have received this email in error
>> please delete it and notify the sender immediately.
>> Before opening any email and/or attachments, please check them for
>> viruses and other defects.
>>
>>
>> ----------------------------------------------------------------------------------------------------------------------------------------------------
>>
>>
>>
>
>
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Brahma Reddy Battula <br...@huawei.com>.
You did not mentioned from when this problem you are facing, I mean after power failure or before that..?
As per my supect after power failure only you are getting this problem where checkpoint was not success..and hostname might have changed after that..
I think, you can try following
i) Change to IP's from hostnames(configurations ) and restart the cluster
ii) Execute savenamespace (Hope you know the usage and steps.)..If this fails we need to do manual steps like following..
we can rename edits.new to edits but this need to check once and confirm by checking code once..
Thanks & Regards
Brahma Reddy Battula
________________________________
From: Stuti Awasthi [stutiawasthi@hcl.com]
Sent: Monday, February 03, 2014 1:25 PM
To: 'user@hadoop.apache.org'
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi Brahma,
Actually for the post , I edited the original IP and hostname in the mail. In real, the hostname(VM-52540048731A) present in /etc/hosts and config files are exactly same
+
You are right “fsck” command is also not working.
hadoop fsck -blocks
Error : 14/02/03 12:52:10 ERROR security.UserGroupInformation: PriviledgedActionException as:hduser cause:java.io.FileNotFoundException: http ://VM-52540048731A:50070/fsck?ugi=hduser&path=%2F&blocks=1
Will replacing IP in place of Hostname will help ?? any other suggestions ?
How to fix the Checkpoint issue also . Directory structure of current and previous.checkpoint directory is below :
stuti@VM-52540048731A:/usr/lib/hadoop/storage/dfs$ ls -l nn/current/
total 8656
-rw-r--r-- 1 hduser hadoop 539 Jan 31 16:50 edits
-rw-r--r-- 1 hduser hadoop 1049088 Feb 3 12:30 edits.new
-rw-r--r-- 1 hduser hadoop 8841839 Jan 31 16:45 fsimage
-rw-r--r-- 1 hduser hadoop 8 Jan 31 16:45 fstime
-rw-r--r-- 1 hduser hadoop 100 Jan 31 16:45 VERSION
stuti@VM-52540048731A:/usr/lib/hadoop/storage/dfs$ ls -l nn/previous.checkpoint/
total 8656
-rw-r--r-- 1 hduser hadoop 539 Jan 31 16:12 edits
-rw-r--r-- 1 hduser hadoop 1049088 Jan 31 16:12 edits.new
-rw-r--r-- 1 hduser hadoop 8841839 Jan 31 16:07 fsimage
-rw-r--r-- 1 hduser hadoop 8 Jan 31 16:07 fstime
-rw-r--r-- 1 hduser hadoop 100 Jan 31 16:07 VERSION
Thanks
Stuti
From: Brahma Reddy Battula [mailto:brahmareddy.battula@huawei.com]
Sent: Monday, February 03, 2014 11:58 AM
To: user@hadoop.apache.org
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi
>From your mail, you are facing following problems.(Mainly NameNode HTTP requests are not going)
i) Checkpoint is not happening
With 1.0.4, only one checkpoint process is executed at a time. When the namenode gets an overlapping checkpointing request, it checks edit.new in its storage directories. If namenode have this file, namenode concludes the previous checkpoint process is not done yet and prints the warning message you've seen.
This is the case if you ensure edits.new file before the error operation residual useless files can be deleted after the detection of whether there is such a problem
In your case While checkpoint was inprogress power failure might occured hence edits.new file was not renamed...Hope before power failure checkpoint was success..
ii) Not able browse the Namenode UI.
I think,,Mostly hostmapping is missconfigured or hostname is changed(Can you cross check hostname you configured in /etc/hosts and hostname of the machine is correct or not).
Why hostname is different from the /etc/hosts to Configuration,..?
>From /etc/hosts/
IP1 Hostname1 # Namenode- vm01 - itself
>From Configurations :
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
In this case ./hadoop fsck / also should not work. can you please try and send us the result..
Thanks & Regards
Brahma Reddy Battula
________________________________
From: Stuti Awasthi [stutiawasthi@hcl.com]
Sent: Monday, February 03, 2014 10:11 AM
To: user
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi All,
Any other tips that can resolve this issue ??
Please suggest
Thanks
Stuti Awasthi
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 8:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
ohh I didn't realized that you are still using 1.0.4 release, yes the property was deprecated in new releases.
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:39 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hadoop version is 1.0.4
In hdfs-default.html for 1.0.4 version we have following property:
dfs.http.address
dfs.secondary.http.address
dfs.namenode.http-address : I suppose this property is not valid for Hadoop 1.x
Please suggest
Thanks
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 7:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010<http://10.139.9.238:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560<tel:4302274560> (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010<http://10.139.9.233:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010<http://10.139.9.232:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010<http://10.139.9.236:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544<tel:8669548544> (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010<http://10.139.9.235:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880<tel:5092986880> (4.74 GB)
Non DFS Used: 8669454336<tel:8669454336> (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010<http://10.139.9.237:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010<http://10.139.9.234:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440<tel:12124221440> (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010<http://10.139.9.231:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<https://email-cn.huawei.com/owa/UrlBlockedError.aspx> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
……..
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Brahma Reddy Battula <br...@huawei.com>.
You did not mentioned from when this problem you are facing, I mean after power failure or before that..?
As per my supect after power failure only you are getting this problem where checkpoint was not success..and hostname might have changed after that..
I think, you can try following
i) Change to IP's from hostnames(configurations ) and restart the cluster
ii) Execute savenamespace (Hope you know the usage and steps.)..If this fails we need to do manual steps like following..
we can rename edits.new to edits but this need to check once and confirm by checking code once..
Thanks & Regards
Brahma Reddy Battula
________________________________
From: Stuti Awasthi [stutiawasthi@hcl.com]
Sent: Monday, February 03, 2014 1:25 PM
To: 'user@hadoop.apache.org'
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi Brahma,
Actually for the post , I edited the original IP and hostname in the mail. In real, the hostname(VM-52540048731A) present in /etc/hosts and config files are exactly same
+
You are right “fsck” command is also not working.
hadoop fsck -blocks
Error : 14/02/03 12:52:10 ERROR security.UserGroupInformation: PriviledgedActionException as:hduser cause:java.io.FileNotFoundException: http ://VM-52540048731A:50070/fsck?ugi=hduser&path=%2F&blocks=1
Will replacing IP in place of Hostname will help ?? any other suggestions ?
How to fix the Checkpoint issue also . Directory structure of current and previous.checkpoint directory is below :
stuti@VM-52540048731A:/usr/lib/hadoop/storage/dfs$ ls -l nn/current/
total 8656
-rw-r--r-- 1 hduser hadoop 539 Jan 31 16:50 edits
-rw-r--r-- 1 hduser hadoop 1049088 Feb 3 12:30 edits.new
-rw-r--r-- 1 hduser hadoop 8841839 Jan 31 16:45 fsimage
-rw-r--r-- 1 hduser hadoop 8 Jan 31 16:45 fstime
-rw-r--r-- 1 hduser hadoop 100 Jan 31 16:45 VERSION
stuti@VM-52540048731A:/usr/lib/hadoop/storage/dfs$ ls -l nn/previous.checkpoint/
total 8656
-rw-r--r-- 1 hduser hadoop 539 Jan 31 16:12 edits
-rw-r--r-- 1 hduser hadoop 1049088 Jan 31 16:12 edits.new
-rw-r--r-- 1 hduser hadoop 8841839 Jan 31 16:07 fsimage
-rw-r--r-- 1 hduser hadoop 8 Jan 31 16:07 fstime
-rw-r--r-- 1 hduser hadoop 100 Jan 31 16:07 VERSION
Thanks
Stuti
From: Brahma Reddy Battula [mailto:brahmareddy.battula@huawei.com]
Sent: Monday, February 03, 2014 11:58 AM
To: user@hadoop.apache.org
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi
>From your mail, you are facing following problems.(Mainly NameNode HTTP requests are not going)
i) Checkpoint is not happening
With 1.0.4, only one checkpoint process is executed at a time. When the namenode gets an overlapping checkpointing request, it checks edit.new in its storage directories. If namenode have this file, namenode concludes the previous checkpoint process is not done yet and prints the warning message you've seen.
This is the case if you ensure edits.new file before the error operation residual useless files can be deleted after the detection of whether there is such a problem
In your case While checkpoint was inprogress power failure might occured hence edits.new file was not renamed...Hope before power failure checkpoint was success..
ii) Not able browse the Namenode UI.
I think,,Mostly hostmapping is missconfigured or hostname is changed(Can you cross check hostname you configured in /etc/hosts and hostname of the machine is correct or not).
Why hostname is different from the /etc/hosts to Configuration,..?
>From /etc/hosts/
IP1 Hostname1 # Namenode- vm01 - itself
>From Configurations :
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
In this case ./hadoop fsck / also should not work. can you please try and send us the result..
Thanks & Regards
Brahma Reddy Battula
________________________________
From: Stuti Awasthi [stutiawasthi@hcl.com]
Sent: Monday, February 03, 2014 10:11 AM
To: user
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi All,
Any other tips that can resolve this issue ??
Please suggest
Thanks
Stuti Awasthi
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 8:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
ohh I didn't realized that you are still using 1.0.4 release, yes the property was deprecated in new releases.
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:39 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hadoop version is 1.0.4
In hdfs-default.html for 1.0.4 version we have following property:
dfs.http.address
dfs.secondary.http.address
dfs.namenode.http-address : I suppose this property is not valid for Hadoop 1.x
Please suggest
Thanks
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 7:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010<http://10.139.9.238:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560<tel:4302274560> (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010<http://10.139.9.233:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010<http://10.139.9.232:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010<http://10.139.9.236:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544<tel:8669548544> (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010<http://10.139.9.235:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880<tel:5092986880> (4.74 GB)
Non DFS Used: 8669454336<tel:8669454336> (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010<http://10.139.9.237:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010<http://10.139.9.234:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440<tel:12124221440> (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010<http://10.139.9.231:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<https://email-cn.huawei.com/owa/UrlBlockedError.aspx> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
……..
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Brahma Reddy Battula <br...@huawei.com>.
You did not mentioned from when this problem you are facing, I mean after power failure or before that..?
As per my supect after power failure only you are getting this problem where checkpoint was not success..and hostname might have changed after that..
I think, you can try following
i) Change to IP's from hostnames(configurations ) and restart the cluster
ii) Execute savenamespace (Hope you know the usage and steps.)..If this fails we need to do manual steps like following..
we can rename edits.new to edits but this need to check once and confirm by checking code once..
Thanks & Regards
Brahma Reddy Battula
________________________________
From: Stuti Awasthi [stutiawasthi@hcl.com]
Sent: Monday, February 03, 2014 1:25 PM
To: 'user@hadoop.apache.org'
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi Brahma,
Actually for the post , I edited the original IP and hostname in the mail. In real, the hostname(VM-52540048731A) present in /etc/hosts and config files are exactly same
+
You are right “fsck” command is also not working.
hadoop fsck -blocks
Error : 14/02/03 12:52:10 ERROR security.UserGroupInformation: PriviledgedActionException as:hduser cause:java.io.FileNotFoundException: http ://VM-52540048731A:50070/fsck?ugi=hduser&path=%2F&blocks=1
Will replacing IP in place of Hostname will help ?? any other suggestions ?
How to fix the Checkpoint issue also . Directory structure of current and previous.checkpoint directory is below :
stuti@VM-52540048731A:/usr/lib/hadoop/storage/dfs$ ls -l nn/current/
total 8656
-rw-r--r-- 1 hduser hadoop 539 Jan 31 16:50 edits
-rw-r--r-- 1 hduser hadoop 1049088 Feb 3 12:30 edits.new
-rw-r--r-- 1 hduser hadoop 8841839 Jan 31 16:45 fsimage
-rw-r--r-- 1 hduser hadoop 8 Jan 31 16:45 fstime
-rw-r--r-- 1 hduser hadoop 100 Jan 31 16:45 VERSION
stuti@VM-52540048731A:/usr/lib/hadoop/storage/dfs$ ls -l nn/previous.checkpoint/
total 8656
-rw-r--r-- 1 hduser hadoop 539 Jan 31 16:12 edits
-rw-r--r-- 1 hduser hadoop 1049088 Jan 31 16:12 edits.new
-rw-r--r-- 1 hduser hadoop 8841839 Jan 31 16:07 fsimage
-rw-r--r-- 1 hduser hadoop 8 Jan 31 16:07 fstime
-rw-r--r-- 1 hduser hadoop 100 Jan 31 16:07 VERSION
Thanks
Stuti
From: Brahma Reddy Battula [mailto:brahmareddy.battula@huawei.com]
Sent: Monday, February 03, 2014 11:58 AM
To: user@hadoop.apache.org
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi
>From your mail, you are facing following problems.(Mainly NameNode HTTP requests are not going)
i) Checkpoint is not happening
With 1.0.4, only one checkpoint process is executed at a time. When the namenode gets an overlapping checkpointing request, it checks edit.new in its storage directories. If namenode have this file, namenode concludes the previous checkpoint process is not done yet and prints the warning message you've seen.
This is the case if you ensure edits.new file before the error operation residual useless files can be deleted after the detection of whether there is such a problem
In your case While checkpoint was inprogress power failure might occured hence edits.new file was not renamed...Hope before power failure checkpoint was success..
ii) Not able browse the Namenode UI.
I think,,Mostly hostmapping is missconfigured or hostname is changed(Can you cross check hostname you configured in /etc/hosts and hostname of the machine is correct or not).
Why hostname is different from the /etc/hosts to Configuration,..?
>From /etc/hosts/
IP1 Hostname1 # Namenode- vm01 - itself
>From Configurations :
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
In this case ./hadoop fsck / also should not work. can you please try and send us the result..
Thanks & Regards
Brahma Reddy Battula
________________________________
From: Stuti Awasthi [stutiawasthi@hcl.com]
Sent: Monday, February 03, 2014 10:11 AM
To: user
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi All,
Any other tips that can resolve this issue ??
Please suggest
Thanks
Stuti Awasthi
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 8:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
ohh I didn't realized that you are still using 1.0.4 release, yes the property was deprecated in new releases.
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:39 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hadoop version is 1.0.4
In hdfs-default.html for 1.0.4 version we have following property:
dfs.http.address
dfs.secondary.http.address
dfs.namenode.http-address : I suppose this property is not valid for Hadoop 1.x
Please suggest
Thanks
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 7:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010<http://10.139.9.238:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560<tel:4302274560> (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010<http://10.139.9.233:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010<http://10.139.9.232:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010<http://10.139.9.236:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544<tel:8669548544> (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010<http://10.139.9.235:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880<tel:5092986880> (4.74 GB)
Non DFS Used: 8669454336<tel:8669454336> (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010<http://10.139.9.237:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010<http://10.139.9.234:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440<tel:12124221440> (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010<http://10.139.9.231:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<https://email-cn.huawei.com/owa/UrlBlockedError.aspx> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
……..
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Brahma Reddy Battula <br...@huawei.com>.
You did not mentioned from when this problem you are facing, I mean after power failure or before that..?
As per my supect after power failure only you are getting this problem where checkpoint was not success..and hostname might have changed after that..
I think, you can try following
i) Change to IP's from hostnames(configurations ) and restart the cluster
ii) Execute savenamespace (Hope you know the usage and steps.)..If this fails we need to do manual steps like following..
we can rename edits.new to edits but this need to check once and confirm by checking code once..
Thanks & Regards
Brahma Reddy Battula
________________________________
From: Stuti Awasthi [stutiawasthi@hcl.com]
Sent: Monday, February 03, 2014 1:25 PM
To: 'user@hadoop.apache.org'
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi Brahma,
Actually for the post , I edited the original IP and hostname in the mail. In real, the hostname(VM-52540048731A) present in /etc/hosts and config files are exactly same
+
You are right “fsck” command is also not working.
hadoop fsck -blocks
Error : 14/02/03 12:52:10 ERROR security.UserGroupInformation: PriviledgedActionException as:hduser cause:java.io.FileNotFoundException: http ://VM-52540048731A:50070/fsck?ugi=hduser&path=%2F&blocks=1
Will replacing IP in place of Hostname will help ?? any other suggestions ?
How to fix the Checkpoint issue also . Directory structure of current and previous.checkpoint directory is below :
stuti@VM-52540048731A:/usr/lib/hadoop/storage/dfs$ ls -l nn/current/
total 8656
-rw-r--r-- 1 hduser hadoop 539 Jan 31 16:50 edits
-rw-r--r-- 1 hduser hadoop 1049088 Feb 3 12:30 edits.new
-rw-r--r-- 1 hduser hadoop 8841839 Jan 31 16:45 fsimage
-rw-r--r-- 1 hduser hadoop 8 Jan 31 16:45 fstime
-rw-r--r-- 1 hduser hadoop 100 Jan 31 16:45 VERSION
stuti@VM-52540048731A:/usr/lib/hadoop/storage/dfs$ ls -l nn/previous.checkpoint/
total 8656
-rw-r--r-- 1 hduser hadoop 539 Jan 31 16:12 edits
-rw-r--r-- 1 hduser hadoop 1049088 Jan 31 16:12 edits.new
-rw-r--r-- 1 hduser hadoop 8841839 Jan 31 16:07 fsimage
-rw-r--r-- 1 hduser hadoop 8 Jan 31 16:07 fstime
-rw-r--r-- 1 hduser hadoop 100 Jan 31 16:07 VERSION
Thanks
Stuti
From: Brahma Reddy Battula [mailto:brahmareddy.battula@huawei.com]
Sent: Monday, February 03, 2014 11:58 AM
To: user@hadoop.apache.org
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi
>From your mail, you are facing following problems.(Mainly NameNode HTTP requests are not going)
i) Checkpoint is not happening
With 1.0.4, only one checkpoint process is executed at a time. When the namenode gets an overlapping checkpointing request, it checks edit.new in its storage directories. If namenode have this file, namenode concludes the previous checkpoint process is not done yet and prints the warning message you've seen.
This is the case if you ensure edits.new file before the error operation residual useless files can be deleted after the detection of whether there is such a problem
In your case While checkpoint was inprogress power failure might occured hence edits.new file was not renamed...Hope before power failure checkpoint was success..
ii) Not able browse the Namenode UI.
I think,,Mostly hostmapping is missconfigured or hostname is changed(Can you cross check hostname you configured in /etc/hosts and hostname of the machine is correct or not).
Why hostname is different from the /etc/hosts to Configuration,..?
>From /etc/hosts/
IP1 Hostname1 # Namenode- vm01 - itself
>From Configurations :
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
In this case ./hadoop fsck / also should not work. can you please try and send us the result..
Thanks & Regards
Brahma Reddy Battula
________________________________
From: Stuti Awasthi [stutiawasthi@hcl.com]
Sent: Monday, February 03, 2014 10:11 AM
To: user
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi All,
Any other tips that can resolve this issue ??
Please suggest
Thanks
Stuti Awasthi
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 8:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
ohh I didn't realized that you are still using 1.0.4 release, yes the property was deprecated in new releases.
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:39 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hadoop version is 1.0.4
In hdfs-default.html for 1.0.4 version we have following property:
dfs.http.address
dfs.secondary.http.address
dfs.namenode.http-address : I suppose this property is not valid for Hadoop 1.x
Please suggest
Thanks
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 7:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010<http://10.139.9.238:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560<tel:4302274560> (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010<http://10.139.9.233:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010<http://10.139.9.232:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010<http://10.139.9.236:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544<tel:8669548544> (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010<http://10.139.9.235:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880<tel:5092986880> (4.74 GB)
Non DFS Used: 8669454336<tel:8669454336> (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010<http://10.139.9.237:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010<http://10.139.9.234:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440<tel:12124221440> (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010<http://10.139.9.231:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<https://email-cn.huawei.com/owa/UrlBlockedError.aspx> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
……..
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Stuti Awasthi <st...@hcl.com>.
Hi Brahma,
Actually for the post , I edited the original IP and hostname in the mail. In real, the hostname(VM-52540048731A) present in /etc/hosts and config files are exactly same
+
You are right "fsck" command is also not working.
hadoop fsck -blocks
Error : 14/02/03 12:52:10 ERROR security.UserGroupInformation: PriviledgedActionException as:hduser cause:java.io.FileNotFoundException: http ://VM-52540048731A:50070/fsck?ugi=hduser&path=%2F&blocks=1
Will replacing IP in place of Hostname will help ?? any other suggestions ?
How to fix the Checkpoint issue also . Directory structure of current and previous.checkpoint directory is below :
stuti@VM-52540048731A:/usr/lib/hadoop/storage/dfs$ ls -l nn/current/
total 8656
-rw-r--r-- 1 hduser hadoop 539 Jan 31 16:50 edits
-rw-r--r-- 1 hduser hadoop 1049088 Feb 3 12:30 edits.new
-rw-r--r-- 1 hduser hadoop 8841839 Jan 31 16:45 fsimage
-rw-r--r-- 1 hduser hadoop 8 Jan 31 16:45 fstime
-rw-r--r-- 1 hduser hadoop 100 Jan 31 16:45 VERSION
stuti@VM-52540048731A:/usr/lib/hadoop/storage/dfs$ ls -l nn/previous.checkpoint/
total 8656
-rw-r--r-- 1 hduser hadoop 539 Jan 31 16:12 edits
-rw-r--r-- 1 hduser hadoop 1049088 Jan 31 16:12 edits.new
-rw-r--r-- 1 hduser hadoop 8841839 Jan 31 16:07 fsimage
-rw-r--r-- 1 hduser hadoop 8 Jan 31 16:07 fstime
-rw-r--r-- 1 hduser hadoop 100 Jan 31 16:07 VERSION
Thanks
Stuti
From: Brahma Reddy Battula [mailto:brahmareddy.battula@huawei.com]
Sent: Monday, February 03, 2014 11:58 AM
To: user@hadoop.apache.org
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi
>From your mail, you are facing following problems.(Mainly NameNode HTTP requests are not going)
i) Checkpoint is not happening
With 1.0.4, only one checkpoint process is executed at a time. When the namenode gets an overlapping checkpointing request, it checks edit.new in its storage directories. If namenode have this file, namenode concludes the previous checkpoint process is not done yet and prints the warning message you've seen.
This is the case if you ensure edits.new file before the error operation residual useless files can be deleted after the detection of whether there is such a problem
In your case While checkpoint was inprogress power failure might occured hence edits.new file was not renamed...Hope before power failure checkpoint was success..
ii) Not able browse the Namenode UI.
I think,,Mostly hostmapping is missconfigured or hostname is changed(Can you cross check hostname you configured in /etc/hosts and hostname of the machine is correct or not).
Why hostname is different from the /etc/hosts to Configuration,..?
>From /etc/hosts/
IP1 Hostname1 # Namenode- vm01 - itself
>From Configurations :
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
In this case ./hadoop fsck / also should not work. can you please try and send us the result..
Thanks & Regards
Brahma Reddy Battula
________________________________
From: Stuti Awasthi [stutiawasthi@hcl.com]
Sent: Monday, February 03, 2014 10:11 AM
To: user
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi All,
Any other tips that can resolve this issue ??
Please suggest
Thanks
Stuti Awasthi
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 8:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
ohh I didn't realized that you are still using 1.0.4 release, yes the property was deprecated in new releases.
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:39 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hadoop version is 1.0.4
In hdfs-default.html for 1.0.4 version we have following property:
dfs.http.address
dfs.secondary.http.address
dfs.namenode.http-address : I suppose this property is not valid for Hadoop 1.x
Please suggest
Thanks
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 7:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010<http://10.139.9.238:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560<tel:4302274560> (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010<http://10.139.9.233:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010<http://10.139.9.232:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010<http://10.139.9.236:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544<tel:8669548544> (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010<http://10.139.9.235:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880<tel:5092986880> (4.74 GB)
Non DFS Used: 8669454336<tel:8669454336> (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010<http://10.139.9.237:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010<http://10.139.9.234:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440<tel:12124221440> (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010<http://10.139.9.231:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<https://email-cn.huawei.com/owa/UrlBlockedError.aspx> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
........
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Stuti Awasthi <st...@hcl.com>.
Hi Brahma,
Actually for the post , I edited the original IP and hostname in the mail. In real, the hostname(VM-52540048731A) present in /etc/hosts and config files are exactly same
+
You are right "fsck" command is also not working.
hadoop fsck -blocks
Error : 14/02/03 12:52:10 ERROR security.UserGroupInformation: PriviledgedActionException as:hduser cause:java.io.FileNotFoundException: http ://VM-52540048731A:50070/fsck?ugi=hduser&path=%2F&blocks=1
Will replacing IP in place of Hostname will help ?? any other suggestions ?
How to fix the Checkpoint issue also . Directory structure of current and previous.checkpoint directory is below :
stuti@VM-52540048731A:/usr/lib/hadoop/storage/dfs$ ls -l nn/current/
total 8656
-rw-r--r-- 1 hduser hadoop 539 Jan 31 16:50 edits
-rw-r--r-- 1 hduser hadoop 1049088 Feb 3 12:30 edits.new
-rw-r--r-- 1 hduser hadoop 8841839 Jan 31 16:45 fsimage
-rw-r--r-- 1 hduser hadoop 8 Jan 31 16:45 fstime
-rw-r--r-- 1 hduser hadoop 100 Jan 31 16:45 VERSION
stuti@VM-52540048731A:/usr/lib/hadoop/storage/dfs$ ls -l nn/previous.checkpoint/
total 8656
-rw-r--r-- 1 hduser hadoop 539 Jan 31 16:12 edits
-rw-r--r-- 1 hduser hadoop 1049088 Jan 31 16:12 edits.new
-rw-r--r-- 1 hduser hadoop 8841839 Jan 31 16:07 fsimage
-rw-r--r-- 1 hduser hadoop 8 Jan 31 16:07 fstime
-rw-r--r-- 1 hduser hadoop 100 Jan 31 16:07 VERSION
Thanks
Stuti
From: Brahma Reddy Battula [mailto:brahmareddy.battula@huawei.com]
Sent: Monday, February 03, 2014 11:58 AM
To: user@hadoop.apache.org
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi
>From your mail, you are facing following problems.(Mainly NameNode HTTP requests are not going)
i) Checkpoint is not happening
With 1.0.4, only one checkpoint process is executed at a time. When the namenode gets an overlapping checkpointing request, it checks edit.new in its storage directories. If namenode have this file, namenode concludes the previous checkpoint process is not done yet and prints the warning message you've seen.
This is the case if you ensure edits.new file before the error operation residual useless files can be deleted after the detection of whether there is such a problem
In your case While checkpoint was inprogress power failure might occured hence edits.new file was not renamed...Hope before power failure checkpoint was success..
ii) Not able browse the Namenode UI.
I think,,Mostly hostmapping is missconfigured or hostname is changed(Can you cross check hostname you configured in /etc/hosts and hostname of the machine is correct or not).
Why hostname is different from the /etc/hosts to Configuration,..?
>From /etc/hosts/
IP1 Hostname1 # Namenode- vm01 - itself
>From Configurations :
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
In this case ./hadoop fsck / also should not work. can you please try and send us the result..
Thanks & Regards
Brahma Reddy Battula
________________________________
From: Stuti Awasthi [stutiawasthi@hcl.com]
Sent: Monday, February 03, 2014 10:11 AM
To: user
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi All,
Any other tips that can resolve this issue ??
Please suggest
Thanks
Stuti Awasthi
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 8:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
ohh I didn't realized that you are still using 1.0.4 release, yes the property was deprecated in new releases.
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:39 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hadoop version is 1.0.4
In hdfs-default.html for 1.0.4 version we have following property:
dfs.http.address
dfs.secondary.http.address
dfs.namenode.http-address : I suppose this property is not valid for Hadoop 1.x
Please suggest
Thanks
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 7:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010<http://10.139.9.238:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560<tel:4302274560> (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010<http://10.139.9.233:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010<http://10.139.9.232:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010<http://10.139.9.236:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544<tel:8669548544> (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010<http://10.139.9.235:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880<tel:5092986880> (4.74 GB)
Non DFS Used: 8669454336<tel:8669454336> (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010<http://10.139.9.237:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010<http://10.139.9.234:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440<tel:12124221440> (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010<http://10.139.9.231:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<https://email-cn.huawei.com/owa/UrlBlockedError.aspx> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
........
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Stuti Awasthi <st...@hcl.com>.
Hi Brahma,
Actually for the post , I edited the original IP and hostname in the mail. In real, the hostname(VM-52540048731A) present in /etc/hosts and config files are exactly same
+
You are right "fsck" command is also not working.
hadoop fsck -blocks
Error : 14/02/03 12:52:10 ERROR security.UserGroupInformation: PriviledgedActionException as:hduser cause:java.io.FileNotFoundException: http ://VM-52540048731A:50070/fsck?ugi=hduser&path=%2F&blocks=1
Will replacing IP in place of Hostname will help ?? any other suggestions ?
How to fix the Checkpoint issue also . Directory structure of current and previous.checkpoint directory is below :
stuti@VM-52540048731A:/usr/lib/hadoop/storage/dfs$ ls -l nn/current/
total 8656
-rw-r--r-- 1 hduser hadoop 539 Jan 31 16:50 edits
-rw-r--r-- 1 hduser hadoop 1049088 Feb 3 12:30 edits.new
-rw-r--r-- 1 hduser hadoop 8841839 Jan 31 16:45 fsimage
-rw-r--r-- 1 hduser hadoop 8 Jan 31 16:45 fstime
-rw-r--r-- 1 hduser hadoop 100 Jan 31 16:45 VERSION
stuti@VM-52540048731A:/usr/lib/hadoop/storage/dfs$ ls -l nn/previous.checkpoint/
total 8656
-rw-r--r-- 1 hduser hadoop 539 Jan 31 16:12 edits
-rw-r--r-- 1 hduser hadoop 1049088 Jan 31 16:12 edits.new
-rw-r--r-- 1 hduser hadoop 8841839 Jan 31 16:07 fsimage
-rw-r--r-- 1 hduser hadoop 8 Jan 31 16:07 fstime
-rw-r--r-- 1 hduser hadoop 100 Jan 31 16:07 VERSION
Thanks
Stuti
From: Brahma Reddy Battula [mailto:brahmareddy.battula@huawei.com]
Sent: Monday, February 03, 2014 11:58 AM
To: user@hadoop.apache.org
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi
>From your mail, you are facing following problems.(Mainly NameNode HTTP requests are not going)
i) Checkpoint is not happening
With 1.0.4, only one checkpoint process is executed at a time. When the namenode gets an overlapping checkpointing request, it checks edit.new in its storage directories. If namenode have this file, namenode concludes the previous checkpoint process is not done yet and prints the warning message you've seen.
This is the case if you ensure edits.new file before the error operation residual useless files can be deleted after the detection of whether there is such a problem
In your case While checkpoint was inprogress power failure might occured hence edits.new file was not renamed...Hope before power failure checkpoint was success..
ii) Not able browse the Namenode UI.
I think,,Mostly hostmapping is missconfigured or hostname is changed(Can you cross check hostname you configured in /etc/hosts and hostname of the machine is correct or not).
Why hostname is different from the /etc/hosts to Configuration,..?
>From /etc/hosts/
IP1 Hostname1 # Namenode- vm01 - itself
>From Configurations :
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
In this case ./hadoop fsck / also should not work. can you please try and send us the result..
Thanks & Regards
Brahma Reddy Battula
________________________________
From: Stuti Awasthi [stutiawasthi@hcl.com]
Sent: Monday, February 03, 2014 10:11 AM
To: user
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi All,
Any other tips that can resolve this issue ??
Please suggest
Thanks
Stuti Awasthi
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 8:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
ohh I didn't realized that you are still using 1.0.4 release, yes the property was deprecated in new releases.
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:39 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hadoop version is 1.0.4
In hdfs-default.html for 1.0.4 version we have following property:
dfs.http.address
dfs.secondary.http.address
dfs.namenode.http-address : I suppose this property is not valid for Hadoop 1.x
Please suggest
Thanks
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 7:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010<http://10.139.9.238:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560<tel:4302274560> (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010<http://10.139.9.233:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010<http://10.139.9.232:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010<http://10.139.9.236:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544<tel:8669548544> (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010<http://10.139.9.235:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880<tel:5092986880> (4.74 GB)
Non DFS Used: 8669454336<tel:8669454336> (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010<http://10.139.9.237:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010<http://10.139.9.234:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440<tel:12124221440> (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010<http://10.139.9.231:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<https://email-cn.huawei.com/owa/UrlBlockedError.aspx> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
........
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Stuti Awasthi <st...@hcl.com>.
Hi Brahma,
Actually for the post , I edited the original IP and hostname in the mail. In real, the hostname(VM-52540048731A) present in /etc/hosts and config files are exactly same
+
You are right "fsck" command is also not working.
hadoop fsck -blocks
Error : 14/02/03 12:52:10 ERROR security.UserGroupInformation: PriviledgedActionException as:hduser cause:java.io.FileNotFoundException: http ://VM-52540048731A:50070/fsck?ugi=hduser&path=%2F&blocks=1
Will replacing IP in place of Hostname will help ?? any other suggestions ?
How to fix the Checkpoint issue also . Directory structure of current and previous.checkpoint directory is below :
stuti@VM-52540048731A:/usr/lib/hadoop/storage/dfs$ ls -l nn/current/
total 8656
-rw-r--r-- 1 hduser hadoop 539 Jan 31 16:50 edits
-rw-r--r-- 1 hduser hadoop 1049088 Feb 3 12:30 edits.new
-rw-r--r-- 1 hduser hadoop 8841839 Jan 31 16:45 fsimage
-rw-r--r-- 1 hduser hadoop 8 Jan 31 16:45 fstime
-rw-r--r-- 1 hduser hadoop 100 Jan 31 16:45 VERSION
stuti@VM-52540048731A:/usr/lib/hadoop/storage/dfs$ ls -l nn/previous.checkpoint/
total 8656
-rw-r--r-- 1 hduser hadoop 539 Jan 31 16:12 edits
-rw-r--r-- 1 hduser hadoop 1049088 Jan 31 16:12 edits.new
-rw-r--r-- 1 hduser hadoop 8841839 Jan 31 16:07 fsimage
-rw-r--r-- 1 hduser hadoop 8 Jan 31 16:07 fstime
-rw-r--r-- 1 hduser hadoop 100 Jan 31 16:07 VERSION
Thanks
Stuti
From: Brahma Reddy Battula [mailto:brahmareddy.battula@huawei.com]
Sent: Monday, February 03, 2014 11:58 AM
To: user@hadoop.apache.org
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi
>From your mail, you are facing following problems.(Mainly NameNode HTTP requests are not going)
i) Checkpoint is not happening
With 1.0.4, only one checkpoint process is executed at a time. When the namenode gets an overlapping checkpointing request, it checks edit.new in its storage directories. If namenode have this file, namenode concludes the previous checkpoint process is not done yet and prints the warning message you've seen.
This is the case if you ensure edits.new file before the error operation residual useless files can be deleted after the detection of whether there is such a problem
In your case While checkpoint was inprogress power failure might occured hence edits.new file was not renamed...Hope before power failure checkpoint was success..
ii) Not able browse the Namenode UI.
I think,,Mostly hostmapping is missconfigured or hostname is changed(Can you cross check hostname you configured in /etc/hosts and hostname of the machine is correct or not).
Why hostname is different from the /etc/hosts to Configuration,..?
>From /etc/hosts/
IP1 Hostname1 # Namenode- vm01 - itself
>From Configurations :
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
In this case ./hadoop fsck / also should not work. can you please try and send us the result..
Thanks & Regards
Brahma Reddy Battula
________________________________
From: Stuti Awasthi [stutiawasthi@hcl.com]
Sent: Monday, February 03, 2014 10:11 AM
To: user
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi All,
Any other tips that can resolve this issue ??
Please suggest
Thanks
Stuti Awasthi
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 8:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
ohh I didn't realized that you are still using 1.0.4 release, yes the property was deprecated in new releases.
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:39 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hadoop version is 1.0.4
In hdfs-default.html for 1.0.4 version we have following property:
dfs.http.address
dfs.secondary.http.address
dfs.namenode.http-address : I suppose this property is not valid for Hadoop 1.x
Please suggest
Thanks
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 7:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010<http://10.139.9.238:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560<tel:4302274560> (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010<http://10.139.9.233:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010<http://10.139.9.232:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010<http://10.139.9.236:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544<tel:8669548544> (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010<http://10.139.9.235:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880<tel:5092986880> (4.74 GB)
Non DFS Used: 8669454336<tel:8669454336> (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010<http://10.139.9.237:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010<http://10.139.9.234:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440<tel:12124221440> (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010<http://10.139.9.231:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<https://email-cn.huawei.com/owa/UrlBlockedError.aspx> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
........
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Brahma Reddy Battula <br...@huawei.com>.
Hi
>From your mail, you are facing following problems.(Mainly NameNode HTTP requests are not going)
i) Checkpoint is not happening
With 1.0.4, only one checkpoint process is executed at a time. When the namenode gets an overlapping checkpointing request, it checks edit.new in its storage directories. If namenode have this file, namenode concludes the previous checkpoint process is not done yet and prints the warning message you've seen.
This is the case if you ensure edits.new file before the error operation residual useless files can be deleted after the detection of whether there is such a problem
In your case While checkpoint was inprogress power failure might occured hence edits.new file was not renamed...Hope before power failure checkpoint was success..
ii) Not able browse the Namenode UI.
I think,,Mostly hostmapping is missconfigured or hostname is changed(Can you cross check hostname you configured in /etc/hosts and hostname of the machine is correct or not).
Why hostname is different from the /etc/hosts to Configuration,..?
>From /etc/hosts/
IP1 Hostname1 # Namenode- vm01 - itself
>From Configurations :
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
In this case ./hadoop fsck / also should not work. can you please try and send us the result..
Thanks & Regards
Brahma Reddy Battula
________________________________
From: Stuti Awasthi [stutiawasthi@hcl.com]
Sent: Monday, February 03, 2014 10:11 AM
To: user
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi All,
Any other tips that can resolve this issue ??
Please suggest
Thanks
Stuti Awasthi
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 8:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
ohh I didn't realized that you are still using 1.0.4 release, yes the property was deprecated in new releases.
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:39 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hadoop version is 1.0.4
In hdfs-default.html for 1.0.4 version we have following property:
dfs.http.address
dfs.secondary.http.address
dfs.namenode.http-address : I suppose this property is not valid for Hadoop 1.x
Please suggest
Thanks
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 7:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010<http://10.139.9.238:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560<tel:4302274560> (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010<http://10.139.9.233:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010<http://10.139.9.232:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010<http://10.139.9.236:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544<tel:8669548544> (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010<http://10.139.9.235:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880<tel:5092986880> (4.74 GB)
Non DFS Used: 8669454336<tel:8669454336> (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010<http://10.139.9.237:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010<http://10.139.9.234:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440<tel:12124221440> (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010<http://10.139.9.231:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<https://email-cn.huawei.com/owa/UrlBlockedError.aspx> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
……..
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Brahma Reddy Battula <br...@huawei.com>.
Hi
>From your mail, you are facing following problems.(Mainly NameNode HTTP requests are not going)
i) Checkpoint is not happening
With 1.0.4, only one checkpoint process is executed at a time. When the namenode gets an overlapping checkpointing request, it checks edit.new in its storage directories. If namenode have this file, namenode concludes the previous checkpoint process is not done yet and prints the warning message you've seen.
This is the case if you ensure edits.new file before the error operation residual useless files can be deleted after the detection of whether there is such a problem
In your case While checkpoint was inprogress power failure might occured hence edits.new file was not renamed...Hope before power failure checkpoint was success..
ii) Not able browse the Namenode UI.
I think,,Mostly hostmapping is missconfigured or hostname is changed(Can you cross check hostname you configured in /etc/hosts and hostname of the machine is correct or not).
Why hostname is different from the /etc/hosts to Configuration,..?
>From /etc/hosts/
IP1 Hostname1 # Namenode- vm01 - itself
>From Configurations :
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
In this case ./hadoop fsck / also should not work. can you please try and send us the result..
Thanks & Regards
Brahma Reddy Battula
________________________________
From: Stuti Awasthi [stutiawasthi@hcl.com]
Sent: Monday, February 03, 2014 10:11 AM
To: user
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi All,
Any other tips that can resolve this issue ??
Please suggest
Thanks
Stuti Awasthi
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 8:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
ohh I didn't realized that you are still using 1.0.4 release, yes the property was deprecated in new releases.
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:39 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hadoop version is 1.0.4
In hdfs-default.html for 1.0.4 version we have following property:
dfs.http.address
dfs.secondary.http.address
dfs.namenode.http-address : I suppose this property is not valid for Hadoop 1.x
Please suggest
Thanks
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 7:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010<http://10.139.9.238:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560<tel:4302274560> (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010<http://10.139.9.233:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010<http://10.139.9.232:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010<http://10.139.9.236:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544<tel:8669548544> (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010<http://10.139.9.235:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880<tel:5092986880> (4.74 GB)
Non DFS Used: 8669454336<tel:8669454336> (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010<http://10.139.9.237:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010<http://10.139.9.234:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440<tel:12124221440> (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010<http://10.139.9.231:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<https://email-cn.huawei.com/owa/UrlBlockedError.aspx> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
……..
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Brahma Reddy Battula <br...@huawei.com>.
Hi
>From your mail, you are facing following problems.(Mainly NameNode HTTP requests are not going)
i) Checkpoint is not happening
With 1.0.4, only one checkpoint process is executed at a time. When the namenode gets an overlapping checkpointing request, it checks edit.new in its storage directories. If namenode have this file, namenode concludes the previous checkpoint process is not done yet and prints the warning message you've seen.
This is the case if you ensure edits.new file before the error operation residual useless files can be deleted after the detection of whether there is such a problem
In your case While checkpoint was inprogress power failure might occured hence edits.new file was not renamed...Hope before power failure checkpoint was success..
ii) Not able browse the Namenode UI.
I think,,Mostly hostmapping is missconfigured or hostname is changed(Can you cross check hostname you configured in /etc/hosts and hostname of the machine is correct or not).
Why hostname is different from the /etc/hosts to Configuration,..?
>From /etc/hosts/
IP1 Hostname1 # Namenode- vm01 - itself
>From Configurations :
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
In this case ./hadoop fsck / also should not work. can you please try and send us the result..
Thanks & Regards
Brahma Reddy Battula
________________________________
From: Stuti Awasthi [stutiawasthi@hcl.com]
Sent: Monday, February 03, 2014 10:11 AM
To: user
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi All,
Any other tips that can resolve this issue ??
Please suggest
Thanks
Stuti Awasthi
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 8:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
ohh I didn't realized that you are still using 1.0.4 release, yes the property was deprecated in new releases.
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:39 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hadoop version is 1.0.4
In hdfs-default.html for 1.0.4 version we have following property:
dfs.http.address
dfs.secondary.http.address
dfs.namenode.http-address : I suppose this property is not valid for Hadoop 1.x
Please suggest
Thanks
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 7:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010<http://10.139.9.238:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560<tel:4302274560> (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010<http://10.139.9.233:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010<http://10.139.9.232:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010<http://10.139.9.236:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544<tel:8669548544> (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010<http://10.139.9.235:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880<tel:5092986880> (4.74 GB)
Non DFS Used: 8669454336<tel:8669454336> (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010<http://10.139.9.237:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010<http://10.139.9.234:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440<tel:12124221440> (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010<http://10.139.9.231:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<https://email-cn.huawei.com/owa/UrlBlockedError.aspx> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
……..
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Brahma Reddy Battula <br...@huawei.com>.
Hi
>From your mail, you are facing following problems.(Mainly NameNode HTTP requests are not going)
i) Checkpoint is not happening
With 1.0.4, only one checkpoint process is executed at a time. When the namenode gets an overlapping checkpointing request, it checks edit.new in its storage directories. If namenode have this file, namenode concludes the previous checkpoint process is not done yet and prints the warning message you've seen.
This is the case if you ensure edits.new file before the error operation residual useless files can be deleted after the detection of whether there is such a problem
In your case While checkpoint was inprogress power failure might occured hence edits.new file was not renamed...Hope before power failure checkpoint was success..
ii) Not able browse the Namenode UI.
I think,,Mostly hostmapping is missconfigured or hostname is changed(Can you cross check hostname you configured in /etc/hosts and hostname of the machine is correct or not).
Why hostname is different from the /etc/hosts to Configuration,..?
>From /etc/hosts/
IP1 Hostname1 # Namenode- vm01 - itself
>From Configurations :
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
In this case ./hadoop fsck / also should not work. can you please try and send us the result..
Thanks & Regards
Brahma Reddy Battula
________________________________
From: Stuti Awasthi [stutiawasthi@hcl.com]
Sent: Monday, February 03, 2014 10:11 AM
To: user
Subject: RE: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi All,
Any other tips that can resolve this issue ??
Please suggest
Thanks
Stuti Awasthi
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 8:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
ohh I didn't realized that you are still using 1.0.4 release, yes the property was deprecated in new releases.
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:39 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hadoop version is 1.0.4
In hdfs-default.html for 1.0.4 version we have following property:
dfs.http.address
dfs.secondary.http.address
dfs.namenode.http-address : I suppose this property is not valid for Hadoop 1.x
Please suggest
Thanks
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 7:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010<http://10.139.9.238:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560<tel:4302274560> (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010<http://10.139.9.233:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010<http://10.139.9.232:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010<http://10.139.9.236:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544<tel:8669548544> (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010<http://10.139.9.235:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880<tel:5092986880> (4.74 GB)
Non DFS Used: 8669454336<tel:8669454336> (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010<http://10.139.9.237:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010<http://10.139.9.234:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440<tel:12124221440> (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010<http://10.139.9.231:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<https://email-cn.huawei.com/owa/UrlBlockedError.aspx> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
……..
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Stuti Awasthi <st...@hcl.com>.
Hi All,
Any other tips that can resolve this issue ??
Please suggest
Thanks
Stuti Awasthi
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 8:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
ohh I didn't realized that you are still using 1.0.4 release, yes the property was deprecated in new releases.
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:39 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hadoop version is 1.0.4
In hdfs-default.html for 1.0.4 version we have following property:
dfs.http.address
dfs.secondary.http.address
dfs.namenode.http-address : I suppose this property is not valid for Hadoop 1.x
Please suggest
Thanks
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 7:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010<http://10.139.9.238:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560<tel:4302274560> (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010<http://10.139.9.233:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010<http://10.139.9.232:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010<http://10.139.9.236:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544<tel:8669548544> (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010<http://10.139.9.235:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880<tel:5092986880> (4.74 GB)
Non DFS Used: 8669454336<tel:8669454336> (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010<http://10.139.9.237:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010<http://10.139.9.234:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440<tel:12124221440> (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010<http://10.139.9.231:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<http://%3cHostname%3e:50070/dfshealth.jsp> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
........
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Stuti Awasthi <st...@hcl.com>.
Hi All,
Any other tips that can resolve this issue ??
Please suggest
Thanks
Stuti Awasthi
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 8:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
ohh I didn't realized that you are still using 1.0.4 release, yes the property was deprecated in new releases.
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:39 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hadoop version is 1.0.4
In hdfs-default.html for 1.0.4 version we have following property:
dfs.http.address
dfs.secondary.http.address
dfs.namenode.http-address : I suppose this property is not valid for Hadoop 1.x
Please suggest
Thanks
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 7:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010<http://10.139.9.238:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560<tel:4302274560> (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010<http://10.139.9.233:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010<http://10.139.9.232:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010<http://10.139.9.236:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544<tel:8669548544> (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010<http://10.139.9.235:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880<tel:5092986880> (4.74 GB)
Non DFS Used: 8669454336<tel:8669454336> (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010<http://10.139.9.237:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010<http://10.139.9.234:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440<tel:12124221440> (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010<http://10.139.9.231:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<http://%3cHostname%3e:50070/dfshealth.jsp> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
........
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Stuti Awasthi <st...@hcl.com>.
Hi All,
Any other tips that can resolve this issue ??
Please suggest
Thanks
Stuti Awasthi
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 8:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
ohh I didn't realized that you are still using 1.0.4 release, yes the property was deprecated in new releases.
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:39 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hadoop version is 1.0.4
In hdfs-default.html for 1.0.4 version we have following property:
dfs.http.address
dfs.secondary.http.address
dfs.namenode.http-address : I suppose this property is not valid for Hadoop 1.x
Please suggest
Thanks
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 7:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010<http://10.139.9.238:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560<tel:4302274560> (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010<http://10.139.9.233:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010<http://10.139.9.232:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010<http://10.139.9.236:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544<tel:8669548544> (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010<http://10.139.9.235:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880<tel:5092986880> (4.74 GB)
Non DFS Used: 8669454336<tel:8669454336> (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010<http://10.139.9.237:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010<http://10.139.9.234:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440<tel:12124221440> (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010<http://10.139.9.231:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<http://%3cHostname%3e:50070/dfshealth.jsp> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
........
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Stuti Awasthi <st...@hcl.com>.
Hi All,
Any other tips that can resolve this issue ??
Please suggest
Thanks
Stuti Awasthi
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 8:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
ohh I didn't realized that you are still using 1.0.4 release, yes the property was deprecated in new releases.
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:39 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hadoop version is 1.0.4
In hdfs-default.html for 1.0.4 version we have following property:
dfs.http.address
dfs.secondary.http.address
dfs.namenode.http-address : I suppose this property is not valid for Hadoop 1.x
Please suggest
Thanks
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 7:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010<http://10.139.9.238:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560<tel:4302274560> (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010<http://10.139.9.233:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010<http://10.139.9.232:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010<http://10.139.9.236:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544<tel:8669548544> (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010<http://10.139.9.235:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880<tel:5092986880> (4.74 GB)
Non DFS Used: 8669454336<tel:8669454336> (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010<http://10.139.9.237:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010<http://10.139.9.234:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440<tel:12124221440> (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010<http://10.139.9.231:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<http://%3cHostname%3e:50070/dfshealth.jsp> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
........
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Posted by Jitendra Yadav <je...@gmail.com>.
ohh I didn't realized that you are still using 1.0.4 release, yes the
property was deprecated in new releases.
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:39 PM, Stuti Awasthi <st...@hcl.com> wrote:
> Hadoop version is 1.0.4
>
> In hdfs-default.html for 1.0.4 version we have following property:
>
> dfs.http.address
>
> dfs.secondary.http.address
>
>
>
>
>
> dfs.namenode.http-address : I suppose this property is not valid for
> Hadoop 1.x
>
>
>
> Please suggest
>
>
>
> Thanks
>
> *From:* Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
> *Sent:* Friday, January 31, 2014 7:26 PM
>
> *To:* user
> *Subject:* Re: java.io.FileNotFoundException:
> http://HOSTNAME:50070/getimage?getimage=1
>
>
>
> Can you please change below property and restart your cluster again?
>
>
>
> FROM:
>
> <name>dfs.http.address</name>
>
>
>
> TO:
>
> <name>dfs.namenode.http-addres</name>
>
>
>
> Thanks
>
> Jitendra
>
>
>
> On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>
> wrote:
>
> Hi Jitendra,
>
>
>
> I realized that some days back ,my cluster was down due to power failure
> after which nn/current directory has : edits, edits.new file and now SNN is
> not rolling these edits due to HTTP error.
>
> Also currently my NN and SNN are operating on same machine
>
>
>
>
>
> DFSadmin report :
>
>
>
> Configured Capacity: 659494076416 (614.2 GB)
>
> Present Capacity: 535599210496 (498.82 GB)
>
> DFS Remaining: 497454006272 (463.29 GB)
>
> DFS Used: 38145204224 (35.53 GB)
>
> DFS Used%: 7.12%
>
> Under replicated blocks: 283
>
> Blocks with corrupt replicas: 3
>
> Missing blocks: 3
>
>
>
> -------------------------------------------------
>
> Datanodes available: 8 (8 total, 0 dead)
>
>
>
> Name: 10.139.9.238:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4302274560 (4.01 GB)
>
> Non DFS Used: 8391843840 (7.82 GB)
>
> DFS Remaining: 69742641152(64.95 GB)
>
> DFS Used%: 5.22%
>
> DFS Remaining%: 84.6%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.233:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 5774745600 (5.38 GB)
>
> Non DFS Used: 13409488896 (12.49 GB)
>
> DFS Remaining: 63252525056(58.91 GB)
>
> DFS Used%: 7.01%
>
> DFS Remaining%: 76.73%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.232:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 8524451840 (7.94 GB)
>
> Non DFS Used: 24847884288 (23.14 GB)
>
> DFS Remaining: 49064423424(45.69 GB)
>
> DFS Used%: 10.34%
>
> DFS Remaining%: 59.52%
>
> Last contact: Fri Jan 31 18:55:21 IST 2014
>
>
>
>
>
> Name: 10.139.9.236:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4543819776 (4.23 GB)
>
> Non DFS Used: 8669548544 (8.07 GB)
>
> DFS Remaining: 69223391232(64.47 GB)
>
> DFS Used%: 5.51%
>
> DFS Remaining%: 83.97%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.235:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 5092986880 (4.74 GB)
>
> Non DFS Used: 8669454336 (8.07 GB)
>
> DFS Remaining: 68674318336(63.96 GB)
>
> DFS Used%: 6.18%
>
> DFS Remaining%: 83.31%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.237:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4604301312 (4.29 GB)
>
> Non DFS Used: 11005788160 (10.25 GB)
>
> DFS Remaining: 66826670080(62.24 GB)
>
> DFS Used%: 5.59%
>
> DFS Remaining%: 81.06%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.234:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4277760000 (3.98 GB)
>
> Non DFS Used: 12124221440 (11.29 GB)
>
> DFS Remaining: 66034778112(61.5 GB)
>
> DFS Used%: 5.19%
>
> DFS Remaining%: 80.1%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.231:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 1024864256 (977.39 MB)
>
> Non DFS Used: 36776636416 (34.25 GB)
>
> DFS Remaining: 44635258880(41.57 GB)
>
> DFS Used%: 1.24%
>
> DFS Remaining%: 54.14%
>
> Last contact: Fri Jan 31 18:55:20 IST 2014
>
>
>
>
>
>
>
> *From:* Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
> *Sent:* Friday, January 31, 2014 6:58 PM
> *To:* user
> *Subject:* Re: java.io.FileNotFoundException:
> http://HOSTNAME:50070/getimage?getimage=1
>
>
>
> Hi,
>
>
>
> Please post the output of dfs report command, this could help us to
> understand cluster health.
>
>
>
> # *hadoop dfsadmin -report*
>
>
>
> Thanks
>
> Jitendra
>
>
>
> On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>
> wrote:
>
> Hi All,
>
>
>
> I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP
> request at port 50070 on dfs is not working properly.
>
> I have an Hadoop cluster which is operating from several days. Recently we
> are also not able to see dfshealth.jsp page from webconsole.
>
>
>
> Problems :
>
> 1. http://<Hostname>:50070/dfshealth.jsp shows following error
>
>
>
> HTTP ERROR: 404
>
> Problem accessing /. Reason:
>
> NOT_FOUND
>
>
>
> 2. SNN is not able to roll edits :
>
> ERROR in SecondaryNameNode Log
>
> java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
>
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
>
> at
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:416)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
>
>
>
> ERROR in Namenode Log
>
> 2014-01-31 18:15:12,046 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 10.139.9.231
>
> 2014-01-31 18:15:12,046 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log,
> edits.new files already exists in all healthy directories:
>
> /usr/lib/hadoop/storage/dfs/nn/current/edits.new
>
>
>
>
>
>
>
> Namenode logs which suggest that webserver is started on 50070
> successfully:
>
> 2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1. Opening the listener on 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50070
> webServer.getConnectors()[0].getLocalPort() returned 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50070
>
> 2014-01-31 14:42:35,378 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
> HOSTNAME:50070
>
>
>
>
>
> Hdfs-site.xml
>
> <configuration>
>
> <property>
>
> <name>dfs.replication</name>
>
> <value>2</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.name.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/nn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.data.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/dn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.permissions</name>
>
> <value>false</value>
>
> </property>
>
> <property>
>
> <name>dfs.webhdfs.enabled</name>
>
> <value>true</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.http.address</name>
>
> <value>HOSTNAME:50070</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.secondary.http.address</name>
>
> <value>HOSTNAME:50090</value>
>
> </property>
>
>
>
> <property>
>
> <name>fs.checkpoint.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/snn</value>
>
> </property>
>
>
>
> </configuration>
>
>
>
>
>
> */etc/hosts* (Note I have also tried by commenting 127.0.0.1 entry in
> host file but the issue was not resolved)
>
>
>
> 127.0.0.1 localhost
>
>
>
> IP1 Hostname1 # Namenode- vm01 - itself
>
> IP2 Hostname2 # DataNode- vm02
>
> ........
>
>
>
> # The following lines are desirable for IPv6 capable hosts
>
> ::1 ip6-localhost ip6-loopback
>
> fe00::0 ip6-localnet
>
> ff00::0 ip6-mcastprefix
>
> ff02::1 ip6-allnodes
>
> ff02::2 ip6-allrouters
>
>
>
>
>
> Note : All Hadoop daemons are executing fine and the jobs are running
> properly.
>
>
>
> How to resolve this issue, I have tried many options provided on different
> forums but still facing the same issue.
>
> I belive that this can cause a major problem later as my edits are not
> getting rolled into fsimage.. This can cause me a data loss in case of
> failure.
>
>
>
> Please suggest
>
>
>
> Thanks
>
> Stuti
>
>
>
>
>
>
>
>
>
> ::DISCLAIMER::
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> E-mail transmission is not guaranteed to be secure or error-free as
> information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents
> (with or without referred errors) shall therefore not attach any liability
> on the originator or HCL or its affiliates.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the
> views or opinions of HCL or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification,
> distribution and / or publication of this message without the prior
> written consent of authorized representative of
> HCL is strictly prohibited. If you have received this email in error
> please delete it and notify the sender immediately.
> Before opening any email and/or attachments, please check them for viruses
> and other defects.
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
>
>
>
>
Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Posted by Jitendra Yadav <je...@gmail.com>.
ohh I didn't realized that you are still using 1.0.4 release, yes the
property was deprecated in new releases.
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:39 PM, Stuti Awasthi <st...@hcl.com> wrote:
> Hadoop version is 1.0.4
>
> In hdfs-default.html for 1.0.4 version we have following property:
>
> dfs.http.address
>
> dfs.secondary.http.address
>
>
>
>
>
> dfs.namenode.http-address : I suppose this property is not valid for
> Hadoop 1.x
>
>
>
> Please suggest
>
>
>
> Thanks
>
> *From:* Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
> *Sent:* Friday, January 31, 2014 7:26 PM
>
> *To:* user
> *Subject:* Re: java.io.FileNotFoundException:
> http://HOSTNAME:50070/getimage?getimage=1
>
>
>
> Can you please change below property and restart your cluster again?
>
>
>
> FROM:
>
> <name>dfs.http.address</name>
>
>
>
> TO:
>
> <name>dfs.namenode.http-addres</name>
>
>
>
> Thanks
>
> Jitendra
>
>
>
> On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>
> wrote:
>
> Hi Jitendra,
>
>
>
> I realized that some days back ,my cluster was down due to power failure
> after which nn/current directory has : edits, edits.new file and now SNN is
> not rolling these edits due to HTTP error.
>
> Also currently my NN and SNN are operating on same machine
>
>
>
>
>
> DFSadmin report :
>
>
>
> Configured Capacity: 659494076416 (614.2 GB)
>
> Present Capacity: 535599210496 (498.82 GB)
>
> DFS Remaining: 497454006272 (463.29 GB)
>
> DFS Used: 38145204224 (35.53 GB)
>
> DFS Used%: 7.12%
>
> Under replicated blocks: 283
>
> Blocks with corrupt replicas: 3
>
> Missing blocks: 3
>
>
>
> -------------------------------------------------
>
> Datanodes available: 8 (8 total, 0 dead)
>
>
>
> Name: 10.139.9.238:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4302274560 (4.01 GB)
>
> Non DFS Used: 8391843840 (7.82 GB)
>
> DFS Remaining: 69742641152(64.95 GB)
>
> DFS Used%: 5.22%
>
> DFS Remaining%: 84.6%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.233:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 5774745600 (5.38 GB)
>
> Non DFS Used: 13409488896 (12.49 GB)
>
> DFS Remaining: 63252525056(58.91 GB)
>
> DFS Used%: 7.01%
>
> DFS Remaining%: 76.73%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.232:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 8524451840 (7.94 GB)
>
> Non DFS Used: 24847884288 (23.14 GB)
>
> DFS Remaining: 49064423424(45.69 GB)
>
> DFS Used%: 10.34%
>
> DFS Remaining%: 59.52%
>
> Last contact: Fri Jan 31 18:55:21 IST 2014
>
>
>
>
>
> Name: 10.139.9.236:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4543819776 (4.23 GB)
>
> Non DFS Used: 8669548544 (8.07 GB)
>
> DFS Remaining: 69223391232(64.47 GB)
>
> DFS Used%: 5.51%
>
> DFS Remaining%: 83.97%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.235:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 5092986880 (4.74 GB)
>
> Non DFS Used: 8669454336 (8.07 GB)
>
> DFS Remaining: 68674318336(63.96 GB)
>
> DFS Used%: 6.18%
>
> DFS Remaining%: 83.31%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.237:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4604301312 (4.29 GB)
>
> Non DFS Used: 11005788160 (10.25 GB)
>
> DFS Remaining: 66826670080(62.24 GB)
>
> DFS Used%: 5.59%
>
> DFS Remaining%: 81.06%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.234:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4277760000 (3.98 GB)
>
> Non DFS Used: 12124221440 (11.29 GB)
>
> DFS Remaining: 66034778112(61.5 GB)
>
> DFS Used%: 5.19%
>
> DFS Remaining%: 80.1%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.231:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 1024864256 (977.39 MB)
>
> Non DFS Used: 36776636416 (34.25 GB)
>
> DFS Remaining: 44635258880(41.57 GB)
>
> DFS Used%: 1.24%
>
> DFS Remaining%: 54.14%
>
> Last contact: Fri Jan 31 18:55:20 IST 2014
>
>
>
>
>
>
>
> *From:* Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
> *Sent:* Friday, January 31, 2014 6:58 PM
> *To:* user
> *Subject:* Re: java.io.FileNotFoundException:
> http://HOSTNAME:50070/getimage?getimage=1
>
>
>
> Hi,
>
>
>
> Please post the output of dfs report command, this could help us to
> understand cluster health.
>
>
>
> # *hadoop dfsadmin -report*
>
>
>
> Thanks
>
> Jitendra
>
>
>
> On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>
> wrote:
>
> Hi All,
>
>
>
> I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP
> request at port 50070 on dfs is not working properly.
>
> I have an Hadoop cluster which is operating from several days. Recently we
> are also not able to see dfshealth.jsp page from webconsole.
>
>
>
> Problems :
>
> 1. http://<Hostname>:50070/dfshealth.jsp shows following error
>
>
>
> HTTP ERROR: 404
>
> Problem accessing /. Reason:
>
> NOT_FOUND
>
>
>
> 2. SNN is not able to roll edits :
>
> ERROR in SecondaryNameNode Log
>
> java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
>
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
>
> at
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:416)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
>
>
>
> ERROR in Namenode Log
>
> 2014-01-31 18:15:12,046 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 10.139.9.231
>
> 2014-01-31 18:15:12,046 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log,
> edits.new files already exists in all healthy directories:
>
> /usr/lib/hadoop/storage/dfs/nn/current/edits.new
>
>
>
>
>
>
>
> Namenode logs which suggest that webserver is started on 50070
> successfully:
>
> 2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1. Opening the listener on 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50070
> webServer.getConnectors()[0].getLocalPort() returned 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50070
>
> 2014-01-31 14:42:35,378 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
> HOSTNAME:50070
>
>
>
>
>
> Hdfs-site.xml
>
> <configuration>
>
> <property>
>
> <name>dfs.replication</name>
>
> <value>2</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.name.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/nn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.data.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/dn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.permissions</name>
>
> <value>false</value>
>
> </property>
>
> <property>
>
> <name>dfs.webhdfs.enabled</name>
>
> <value>true</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.http.address</name>
>
> <value>HOSTNAME:50070</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.secondary.http.address</name>
>
> <value>HOSTNAME:50090</value>
>
> </property>
>
>
>
> <property>
>
> <name>fs.checkpoint.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/snn</value>
>
> </property>
>
>
>
> </configuration>
>
>
>
>
>
> */etc/hosts* (Note I have also tried by commenting 127.0.0.1 entry in
> host file but the issue was not resolved)
>
>
>
> 127.0.0.1 localhost
>
>
>
> IP1 Hostname1 # Namenode- vm01 - itself
>
> IP2 Hostname2 # DataNode- vm02
>
> ........
>
>
>
> # The following lines are desirable for IPv6 capable hosts
>
> ::1 ip6-localhost ip6-loopback
>
> fe00::0 ip6-localnet
>
> ff00::0 ip6-mcastprefix
>
> ff02::1 ip6-allnodes
>
> ff02::2 ip6-allrouters
>
>
>
>
>
> Note : All Hadoop daemons are executing fine and the jobs are running
> properly.
>
>
>
> How to resolve this issue, I have tried many options provided on different
> forums but still facing the same issue.
>
> I belive that this can cause a major problem later as my edits are not
> getting rolled into fsimage.. This can cause me a data loss in case of
> failure.
>
>
>
> Please suggest
>
>
>
> Thanks
>
> Stuti
>
>
>
>
>
>
>
>
>
> ::DISCLAIMER::
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> E-mail transmission is not guaranteed to be secure or error-free as
> information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents
> (with or without referred errors) shall therefore not attach any liability
> on the originator or HCL or its affiliates.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the
> views or opinions of HCL or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification,
> distribution and / or publication of this message without the prior
> written consent of authorized representative of
> HCL is strictly prohibited. If you have received this email in error
> please delete it and notify the sender immediately.
> Before opening any email and/or attachments, please check them for viruses
> and other defects.
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
>
>
>
>
Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Posted by Jitendra Yadav <je...@gmail.com>.
ohh I didn't realized that you are still using 1.0.4 release, yes the
property was deprecated in new releases.
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:39 PM, Stuti Awasthi <st...@hcl.com> wrote:
> Hadoop version is 1.0.4
>
> In hdfs-default.html for 1.0.4 version we have following property:
>
> dfs.http.address
>
> dfs.secondary.http.address
>
>
>
>
>
> dfs.namenode.http-address : I suppose this property is not valid for
> Hadoop 1.x
>
>
>
> Please suggest
>
>
>
> Thanks
>
> *From:* Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
> *Sent:* Friday, January 31, 2014 7:26 PM
>
> *To:* user
> *Subject:* Re: java.io.FileNotFoundException:
> http://HOSTNAME:50070/getimage?getimage=1
>
>
>
> Can you please change below property and restart your cluster again?
>
>
>
> FROM:
>
> <name>dfs.http.address</name>
>
>
>
> TO:
>
> <name>dfs.namenode.http-addres</name>
>
>
>
> Thanks
>
> Jitendra
>
>
>
> On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>
> wrote:
>
> Hi Jitendra,
>
>
>
> I realized that some days back ,my cluster was down due to power failure
> after which nn/current directory has : edits, edits.new file and now SNN is
> not rolling these edits due to HTTP error.
>
> Also currently my NN and SNN are operating on same machine
>
>
>
>
>
> DFSadmin report :
>
>
>
> Configured Capacity: 659494076416 (614.2 GB)
>
> Present Capacity: 535599210496 (498.82 GB)
>
> DFS Remaining: 497454006272 (463.29 GB)
>
> DFS Used: 38145204224 (35.53 GB)
>
> DFS Used%: 7.12%
>
> Under replicated blocks: 283
>
> Blocks with corrupt replicas: 3
>
> Missing blocks: 3
>
>
>
> -------------------------------------------------
>
> Datanodes available: 8 (8 total, 0 dead)
>
>
>
> Name: 10.139.9.238:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4302274560 (4.01 GB)
>
> Non DFS Used: 8391843840 (7.82 GB)
>
> DFS Remaining: 69742641152(64.95 GB)
>
> DFS Used%: 5.22%
>
> DFS Remaining%: 84.6%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.233:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 5774745600 (5.38 GB)
>
> Non DFS Used: 13409488896 (12.49 GB)
>
> DFS Remaining: 63252525056(58.91 GB)
>
> DFS Used%: 7.01%
>
> DFS Remaining%: 76.73%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.232:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 8524451840 (7.94 GB)
>
> Non DFS Used: 24847884288 (23.14 GB)
>
> DFS Remaining: 49064423424(45.69 GB)
>
> DFS Used%: 10.34%
>
> DFS Remaining%: 59.52%
>
> Last contact: Fri Jan 31 18:55:21 IST 2014
>
>
>
>
>
> Name: 10.139.9.236:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4543819776 (4.23 GB)
>
> Non DFS Used: 8669548544 (8.07 GB)
>
> DFS Remaining: 69223391232(64.47 GB)
>
> DFS Used%: 5.51%
>
> DFS Remaining%: 83.97%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.235:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 5092986880 (4.74 GB)
>
> Non DFS Used: 8669454336 (8.07 GB)
>
> DFS Remaining: 68674318336(63.96 GB)
>
> DFS Used%: 6.18%
>
> DFS Remaining%: 83.31%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.237:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4604301312 (4.29 GB)
>
> Non DFS Used: 11005788160 (10.25 GB)
>
> DFS Remaining: 66826670080(62.24 GB)
>
> DFS Used%: 5.59%
>
> DFS Remaining%: 81.06%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.234:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4277760000 (3.98 GB)
>
> Non DFS Used: 12124221440 (11.29 GB)
>
> DFS Remaining: 66034778112(61.5 GB)
>
> DFS Used%: 5.19%
>
> DFS Remaining%: 80.1%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.231:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 1024864256 (977.39 MB)
>
> Non DFS Used: 36776636416 (34.25 GB)
>
> DFS Remaining: 44635258880(41.57 GB)
>
> DFS Used%: 1.24%
>
> DFS Remaining%: 54.14%
>
> Last contact: Fri Jan 31 18:55:20 IST 2014
>
>
>
>
>
>
>
> *From:* Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
> *Sent:* Friday, January 31, 2014 6:58 PM
> *To:* user
> *Subject:* Re: java.io.FileNotFoundException:
> http://HOSTNAME:50070/getimage?getimage=1
>
>
>
> Hi,
>
>
>
> Please post the output of dfs report command, this could help us to
> understand cluster health.
>
>
>
> # *hadoop dfsadmin -report*
>
>
>
> Thanks
>
> Jitendra
>
>
>
> On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>
> wrote:
>
> Hi All,
>
>
>
> I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP
> request at port 50070 on dfs is not working properly.
>
> I have an Hadoop cluster which is operating from several days. Recently we
> are also not able to see dfshealth.jsp page from webconsole.
>
>
>
> Problems :
>
> 1. http://<Hostname>:50070/dfshealth.jsp shows following error
>
>
>
> HTTP ERROR: 404
>
> Problem accessing /. Reason:
>
> NOT_FOUND
>
>
>
> 2. SNN is not able to roll edits :
>
> ERROR in SecondaryNameNode Log
>
> java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
>
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
>
> at
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:416)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
>
>
>
> ERROR in Namenode Log
>
> 2014-01-31 18:15:12,046 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 10.139.9.231
>
> 2014-01-31 18:15:12,046 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log,
> edits.new files already exists in all healthy directories:
>
> /usr/lib/hadoop/storage/dfs/nn/current/edits.new
>
>
>
>
>
>
>
> Namenode logs which suggest that webserver is started on 50070
> successfully:
>
> 2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1. Opening the listener on 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50070
> webServer.getConnectors()[0].getLocalPort() returned 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50070
>
> 2014-01-31 14:42:35,378 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
> HOSTNAME:50070
>
>
>
>
>
> Hdfs-site.xml
>
> <configuration>
>
> <property>
>
> <name>dfs.replication</name>
>
> <value>2</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.name.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/nn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.data.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/dn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.permissions</name>
>
> <value>false</value>
>
> </property>
>
> <property>
>
> <name>dfs.webhdfs.enabled</name>
>
> <value>true</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.http.address</name>
>
> <value>HOSTNAME:50070</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.secondary.http.address</name>
>
> <value>HOSTNAME:50090</value>
>
> </property>
>
>
>
> <property>
>
> <name>fs.checkpoint.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/snn</value>
>
> </property>
>
>
>
> </configuration>
>
>
>
>
>
> */etc/hosts* (Note I have also tried by commenting 127.0.0.1 entry in
> host file but the issue was not resolved)
>
>
>
> 127.0.0.1 localhost
>
>
>
> IP1 Hostname1 # Namenode- vm01 - itself
>
> IP2 Hostname2 # DataNode- vm02
>
> ........
>
>
>
> # The following lines are desirable for IPv6 capable hosts
>
> ::1 ip6-localhost ip6-loopback
>
> fe00::0 ip6-localnet
>
> ff00::0 ip6-mcastprefix
>
> ff02::1 ip6-allnodes
>
> ff02::2 ip6-allrouters
>
>
>
>
>
> Note : All Hadoop daemons are executing fine and the jobs are running
> properly.
>
>
>
> How to resolve this issue, I have tried many options provided on different
> forums but still facing the same issue.
>
> I belive that this can cause a major problem later as my edits are not
> getting rolled into fsimage.. This can cause me a data loss in case of
> failure.
>
>
>
> Please suggest
>
>
>
> Thanks
>
> Stuti
>
>
>
>
>
>
>
>
>
> ::DISCLAIMER::
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> E-mail transmission is not guaranteed to be secure or error-free as
> information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents
> (with or without referred errors) shall therefore not attach any liability
> on the originator or HCL or its affiliates.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the
> views or opinions of HCL or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification,
> distribution and / or publication of this message without the prior
> written consent of authorized representative of
> HCL is strictly prohibited. If you have received this email in error
> please delete it and notify the sender immediately.
> Before opening any email and/or attachments, please check them for viruses
> and other defects.
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
>
>
>
>
Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Posted by Jitendra Yadav <je...@gmail.com>.
ohh I didn't realized that you are still using 1.0.4 release, yes the
property was deprecated in new releases.
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:39 PM, Stuti Awasthi <st...@hcl.com> wrote:
> Hadoop version is 1.0.4
>
> In hdfs-default.html for 1.0.4 version we have following property:
>
> dfs.http.address
>
> dfs.secondary.http.address
>
>
>
>
>
> dfs.namenode.http-address : I suppose this property is not valid for
> Hadoop 1.x
>
>
>
> Please suggest
>
>
>
> Thanks
>
> *From:* Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
> *Sent:* Friday, January 31, 2014 7:26 PM
>
> *To:* user
> *Subject:* Re: java.io.FileNotFoundException:
> http://HOSTNAME:50070/getimage?getimage=1
>
>
>
> Can you please change below property and restart your cluster again?
>
>
>
> FROM:
>
> <name>dfs.http.address</name>
>
>
>
> TO:
>
> <name>dfs.namenode.http-addres</name>
>
>
>
> Thanks
>
> Jitendra
>
>
>
> On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>
> wrote:
>
> Hi Jitendra,
>
>
>
> I realized that some days back ,my cluster was down due to power failure
> after which nn/current directory has : edits, edits.new file and now SNN is
> not rolling these edits due to HTTP error.
>
> Also currently my NN and SNN are operating on same machine
>
>
>
>
>
> DFSadmin report :
>
>
>
> Configured Capacity: 659494076416 (614.2 GB)
>
> Present Capacity: 535599210496 (498.82 GB)
>
> DFS Remaining: 497454006272 (463.29 GB)
>
> DFS Used: 38145204224 (35.53 GB)
>
> DFS Used%: 7.12%
>
> Under replicated blocks: 283
>
> Blocks with corrupt replicas: 3
>
> Missing blocks: 3
>
>
>
> -------------------------------------------------
>
> Datanodes available: 8 (8 total, 0 dead)
>
>
>
> Name: 10.139.9.238:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4302274560 (4.01 GB)
>
> Non DFS Used: 8391843840 (7.82 GB)
>
> DFS Remaining: 69742641152(64.95 GB)
>
> DFS Used%: 5.22%
>
> DFS Remaining%: 84.6%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.233:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 5774745600 (5.38 GB)
>
> Non DFS Used: 13409488896 (12.49 GB)
>
> DFS Remaining: 63252525056(58.91 GB)
>
> DFS Used%: 7.01%
>
> DFS Remaining%: 76.73%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.232:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 8524451840 (7.94 GB)
>
> Non DFS Used: 24847884288 (23.14 GB)
>
> DFS Remaining: 49064423424(45.69 GB)
>
> DFS Used%: 10.34%
>
> DFS Remaining%: 59.52%
>
> Last contact: Fri Jan 31 18:55:21 IST 2014
>
>
>
>
>
> Name: 10.139.9.236:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4543819776 (4.23 GB)
>
> Non DFS Used: 8669548544 (8.07 GB)
>
> DFS Remaining: 69223391232(64.47 GB)
>
> DFS Used%: 5.51%
>
> DFS Remaining%: 83.97%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.235:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 5092986880 (4.74 GB)
>
> Non DFS Used: 8669454336 (8.07 GB)
>
> DFS Remaining: 68674318336(63.96 GB)
>
> DFS Used%: 6.18%
>
> DFS Remaining%: 83.31%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.237:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4604301312 (4.29 GB)
>
> Non DFS Used: 11005788160 (10.25 GB)
>
> DFS Remaining: 66826670080(62.24 GB)
>
> DFS Used%: 5.59%
>
> DFS Remaining%: 81.06%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.234:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4277760000 (3.98 GB)
>
> Non DFS Used: 12124221440 (11.29 GB)
>
> DFS Remaining: 66034778112(61.5 GB)
>
> DFS Used%: 5.19%
>
> DFS Remaining%: 80.1%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.231:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 1024864256 (977.39 MB)
>
> Non DFS Used: 36776636416 (34.25 GB)
>
> DFS Remaining: 44635258880(41.57 GB)
>
> DFS Used%: 1.24%
>
> DFS Remaining%: 54.14%
>
> Last contact: Fri Jan 31 18:55:20 IST 2014
>
>
>
>
>
>
>
> *From:* Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
> *Sent:* Friday, January 31, 2014 6:58 PM
> *To:* user
> *Subject:* Re: java.io.FileNotFoundException:
> http://HOSTNAME:50070/getimage?getimage=1
>
>
>
> Hi,
>
>
>
> Please post the output of dfs report command, this could help us to
> understand cluster health.
>
>
>
> # *hadoop dfsadmin -report*
>
>
>
> Thanks
>
> Jitendra
>
>
>
> On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>
> wrote:
>
> Hi All,
>
>
>
> I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP
> request at port 50070 on dfs is not working properly.
>
> I have an Hadoop cluster which is operating from several days. Recently we
> are also not able to see dfshealth.jsp page from webconsole.
>
>
>
> Problems :
>
> 1. http://<Hostname>:50070/dfshealth.jsp shows following error
>
>
>
> HTTP ERROR: 404
>
> Problem accessing /. Reason:
>
> NOT_FOUND
>
>
>
> 2. SNN is not able to roll edits :
>
> ERROR in SecondaryNameNode Log
>
> java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
>
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
>
> at
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:416)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
>
>
>
> ERROR in Namenode Log
>
> 2014-01-31 18:15:12,046 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 10.139.9.231
>
> 2014-01-31 18:15:12,046 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log,
> edits.new files already exists in all healthy directories:
>
> /usr/lib/hadoop/storage/dfs/nn/current/edits.new
>
>
>
>
>
>
>
> Namenode logs which suggest that webserver is started on 50070
> successfully:
>
> 2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1. Opening the listener on 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50070
> webServer.getConnectors()[0].getLocalPort() returned 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50070
>
> 2014-01-31 14:42:35,378 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
> HOSTNAME:50070
>
>
>
>
>
> Hdfs-site.xml
>
> <configuration>
>
> <property>
>
> <name>dfs.replication</name>
>
> <value>2</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.name.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/nn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.data.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/dn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.permissions</name>
>
> <value>false</value>
>
> </property>
>
> <property>
>
> <name>dfs.webhdfs.enabled</name>
>
> <value>true</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.http.address</name>
>
> <value>HOSTNAME:50070</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.secondary.http.address</name>
>
> <value>HOSTNAME:50090</value>
>
> </property>
>
>
>
> <property>
>
> <name>fs.checkpoint.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/snn</value>
>
> </property>
>
>
>
> </configuration>
>
>
>
>
>
> */etc/hosts* (Note I have also tried by commenting 127.0.0.1 entry in
> host file but the issue was not resolved)
>
>
>
> 127.0.0.1 localhost
>
>
>
> IP1 Hostname1 # Namenode- vm01 - itself
>
> IP2 Hostname2 # DataNode- vm02
>
> ........
>
>
>
> # The following lines are desirable for IPv6 capable hosts
>
> ::1 ip6-localhost ip6-loopback
>
> fe00::0 ip6-localnet
>
> ff00::0 ip6-mcastprefix
>
> ff02::1 ip6-allnodes
>
> ff02::2 ip6-allrouters
>
>
>
>
>
> Note : All Hadoop daemons are executing fine and the jobs are running
> properly.
>
>
>
> How to resolve this issue, I have tried many options provided on different
> forums but still facing the same issue.
>
> I belive that this can cause a major problem later as my edits are not
> getting rolled into fsimage.. This can cause me a data loss in case of
> failure.
>
>
>
> Please suggest
>
>
>
> Thanks
>
> Stuti
>
>
>
>
>
>
>
>
>
> ::DISCLAIMER::
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> E-mail transmission is not guaranteed to be secure or error-free as
> information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents
> (with or without referred errors) shall therefore not attach any liability
> on the originator or HCL or its affiliates.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the
> views or opinions of HCL or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification,
> distribution and / or publication of this message without the prior
> written consent of authorized representative of
> HCL is strictly prohibited. If you have received this email in error
> please delete it and notify the sender immediately.
> Before opening any email and/or attachments, please check them for viruses
> and other defects.
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
>
>
>
>
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Stuti Awasthi <st...@hcl.com>.
Hadoop version is 1.0.4
In hdfs-default.html for 1.0.4 version we have following property:
dfs.http.address
dfs.secondary.http.address
dfs.namenode.http-address : I suppose this property is not valid for Hadoop 1.x
Please suggest
Thanks
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 7:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010<http://10.139.9.238:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560<tel:4302274560> (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010<http://10.139.9.233:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010<http://10.139.9.232:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010<http://10.139.9.236:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544<tel:8669548544> (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010<http://10.139.9.235:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880<tel:5092986880> (4.74 GB)
Non DFS Used: 8669454336<tel:8669454336> (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010<http://10.139.9.237:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010<http://10.139.9.234:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440<tel:12124221440> (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010<http://10.139.9.231:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<http://%3cHostname%3e:50070/dfshealth.jsp> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
........
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Stuti Awasthi <st...@hcl.com>.
Hadoop version is 1.0.4
In hdfs-default.html for 1.0.4 version we have following property:
dfs.http.address
dfs.secondary.http.address
dfs.namenode.http-address : I suppose this property is not valid for Hadoop 1.x
Please suggest
Thanks
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 7:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010<http://10.139.9.238:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560<tel:4302274560> (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010<http://10.139.9.233:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010<http://10.139.9.232:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010<http://10.139.9.236:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544<tel:8669548544> (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010<http://10.139.9.235:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880<tel:5092986880> (4.74 GB)
Non DFS Used: 8669454336<tel:8669454336> (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010<http://10.139.9.237:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010<http://10.139.9.234:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440<tel:12124221440> (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010<http://10.139.9.231:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<http://%3cHostname%3e:50070/dfshealth.jsp> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
........
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Stuti Awasthi <st...@hcl.com>.
Hadoop version is 1.0.4
In hdfs-default.html for 1.0.4 version we have following property:
dfs.http.address
dfs.secondary.http.address
dfs.namenode.http-address : I suppose this property is not valid for Hadoop 1.x
Please suggest
Thanks
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 7:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010<http://10.139.9.238:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560<tel:4302274560> (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010<http://10.139.9.233:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010<http://10.139.9.232:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010<http://10.139.9.236:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544<tel:8669548544> (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010<http://10.139.9.235:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880<tel:5092986880> (4.74 GB)
Non DFS Used: 8669454336<tel:8669454336> (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010<http://10.139.9.237:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010<http://10.139.9.234:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440<tel:12124221440> (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010<http://10.139.9.231:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<http://%3cHostname%3e:50070/dfshealth.jsp> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
........
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Posted by Jitendra Yadav <je...@gmail.com>.
Correcting typo error.
*<name>dfs.namenode.http-address</name>*
*Thanks*
On Fri, Jan 31, 2014 at 7:25 PM, Jitendra Yadav
<je...@gmail.com>wrote:
> Can you please change below property and restart your cluster again?
>
> FROM:
>
> <name>dfs.http.address</name>
>
>
> TO:
> <name>dfs.namenode.http-addres</name>
>
> Thanks
> Jitendra
>
>
> On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>wrote:
>
>> Hi Jitendra,
>>
>>
>>
>> I realized that some days back ,my cluster was down due to power failure
>> after which nn/current directory has : edits, edits.new file and now SNN is
>> not rolling these edits due to HTTP error.
>>
>> Also currently my NN and SNN are operating on same machine
>>
>>
>>
>>
>>
>> DFSadmin report :
>>
>>
>>
>> Configured Capacity: 659494076416 (614.2 GB)
>>
>> Present Capacity: 535599210496 (498.82 GB)
>>
>> DFS Remaining: 497454006272 (463.29 GB)
>>
>> DFS Used: 38145204224 (35.53 GB)
>>
>> DFS Used%: 7.12%
>>
>> Under replicated blocks: 283
>>
>> Blocks with corrupt replicas: 3
>>
>> Missing blocks: 3
>>
>>
>>
>> -------------------------------------------------
>>
>> Datanodes available: 8 (8 total, 0 dead)
>>
>>
>>
>> Name: 10.139.9.238:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 4302274560 (4.01 GB)
>>
>> Non DFS Used: 8391843840 (7.82 GB)
>>
>> DFS Remaining: 69742641152(64.95 GB)
>>
>> DFS Used%: 5.22%
>>
>> DFS Remaining%: 84.6%
>>
>> Last contact: Fri Jan 31 18:55:18 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.233:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 5774745600 (5.38 GB)
>>
>> Non DFS Used: 13409488896 (12.49 GB)
>>
>> DFS Remaining: 63252525056(58.91 GB)
>>
>> DFS Used%: 7.01%
>>
>> DFS Remaining%: 76.73%
>>
>> Last contact: Fri Jan 31 18:55:19 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.232:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 8524451840 (7.94 GB)
>>
>> Non DFS Used: 24847884288 (23.14 GB)
>>
>> DFS Remaining: 49064423424(45.69 GB)
>>
>> DFS Used%: 10.34%
>>
>> DFS Remaining%: 59.52%
>>
>> Last contact: Fri Jan 31 18:55:21 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.236:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 4543819776 (4.23 GB)
>>
>> Non DFS Used: 8669548544 (8.07 GB)
>>
>> DFS Remaining: 69223391232(64.47 GB)
>>
>> DFS Used%: 5.51%
>>
>> DFS Remaining%: 83.97%
>>
>> Last contact: Fri Jan 31 18:55:19 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.235:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 5092986880 (4.74 GB)
>>
>> Non DFS Used: 8669454336 (8.07 GB)
>>
>> DFS Remaining: 68674318336(63.96 GB)
>>
>> DFS Used%: 6.18%
>>
>> DFS Remaining%: 83.31%
>>
>> Last contact: Fri Jan 31 18:55:19 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.237:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 4604301312 (4.29 GB)
>>
>> Non DFS Used: 11005788160 (10.25 GB)
>>
>> DFS Remaining: 66826670080(62.24 GB)
>>
>> DFS Used%: 5.59%
>>
>> DFS Remaining%: 81.06%
>>
>> Last contact: Fri Jan 31 18:55:18 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.234:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 4277760000 (3.98 GB)
>>
>> Non DFS Used: 12124221440 (11.29 GB)
>>
>> DFS Remaining: 66034778112(61.5 GB)
>>
>> DFS Used%: 5.19%
>>
>> DFS Remaining%: 80.1%
>>
>> Last contact: Fri Jan 31 18:55:18 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.231:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 1024864256 (977.39 MB)
>>
>> Non DFS Used: 36776636416 (34.25 GB)
>>
>> DFS Remaining: 44635258880(41.57 GB)
>>
>> DFS Used%: 1.24%
>>
>> DFS Remaining%: 54.14%
>>
>> Last contact: Fri Jan 31 18:55:20 IST 2014
>>
>>
>>
>>
>>
>>
>>
>> *From:* Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
>> *Sent:* Friday, January 31, 2014 6:58 PM
>> *To:* user
>> *Subject:* Re: java.io.FileNotFoundException:
>> http://HOSTNAME:50070/getimage?getimage=1
>>
>>
>>
>> Hi,
>>
>>
>>
>> Please post the output of dfs report command, this could help us to
>> understand cluster health.
>>
>>
>>
>> # *hadoop dfsadmin -report*
>>
>>
>>
>> Thanks
>>
>> Jitendra
>>
>>
>>
>> On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>
>> wrote:
>>
>> Hi All,
>>
>>
>>
>> I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP
>> request at port 50070 on dfs is not working properly.
>>
>> I have an Hadoop cluster which is operating from several days. Recently
>> we are also not able to see dfshealth.jsp page from webconsole.
>>
>>
>>
>> Problems :
>>
>> 1. http://<Hostname>:50070/dfshealth.jsp shows following error
>>
>>
>>
>> HTTP ERROR: 404
>>
>> Problem accessing /. Reason:
>>
>> NOT_FOUND
>>
>>
>>
>> 2. SNN is not able to roll edits :
>>
>> ERROR in SecondaryNameNode Log
>>
>> java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
>>
>> at
>> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
>>
>> at java.security.AccessController.doPrivileged(Native Method)
>>
>> at javax.security.auth.Subject.doAs(Subject.java:416)
>>
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
>>
>>
>>
>> ERROR in Namenode Log
>>
>> 2014-01-31 18:15:12,046 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>> 10.139.9.231
>>
>> 2014-01-31 18:15:12,046 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log,
>> edits.new files already exists in all healthy directories:
>>
>> /usr/lib/hadoop/storage/dfs/nn/current/edits.new
>>
>>
>>
>>
>>
>>
>>
>> Namenode logs which suggest that webserver is started on 50070
>> successfully:
>>
>> 2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port
>> returned by webServer.getConnectors()[0].getLocalPort() before open() is
>> -1. Opening the listener on 50070
>>
>> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer:
>> listener.getLocalPort() returned 50070
>> webServer.getConnectors()[0].getLocalPort() returned 50070
>>
>> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty
>> bound to port 50070
>>
>> 2014-01-31 14:42:35,378 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
>> HOSTNAME:50070
>>
>>
>>
>>
>>
>> Hdfs-site.xml
>>
>> <configuration>
>>
>> <property>
>>
>> <name>dfs.replication</name>
>>
>> <value>2</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>dfs.name.dir</name>
>>
>> <value>/usr/lib/hadoop/storage/dfs/nn</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>dfs.data.dir</name>
>>
>> <value>/usr/lib/hadoop/storage/dfs/dn</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>dfs.permissions</name>
>>
>> <value>false</value>
>>
>> </property>
>>
>> <property>
>>
>> <name>dfs.webhdfs.enabled</name>
>>
>> <value>true</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>dfs.http.address</name>
>>
>> <value>HOSTNAME:50070</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>dfs.secondary.http.address</name>
>>
>> <value>HOSTNAME:50090</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>fs.checkpoint.dir</name>
>>
>> <value>/usr/lib/hadoop/storage/dfs/snn</value>
>>
>> </property>
>>
>>
>>
>> </configuration>
>>
>>
>>
>>
>>
>> */etc/hosts* (Note I have also tried by commenting 127.0.0.1 entry in
>> host file but the issue was not resolved)
>>
>>
>>
>> 127.0.0.1 localhost
>>
>>
>>
>> IP1 Hostname1 # Namenode- vm01 - itself
>>
>> IP2 Hostname2 # DataNode- vm02
>>
>> ........
>>
>>
>>
>> # The following lines are desirable for IPv6 capable hosts
>>
>> ::1 ip6-localhost ip6-loopback
>>
>> fe00::0 ip6-localnet
>>
>> ff00::0 ip6-mcastprefix
>>
>> ff02::1 ip6-allnodes
>>
>> ff02::2 ip6-allrouters
>>
>>
>>
>>
>>
>> Note : All Hadoop daemons are executing fine and the jobs are running
>> properly.
>>
>>
>>
>> How to resolve this issue, I have tried many options provided on
>> different forums but still facing the same issue.
>>
>> I belive that this can cause a major problem later as my edits are not
>> getting rolled into fsimage.. This can cause me a data loss in case of
>> failure.
>>
>>
>>
>> Please suggest
>>
>>
>>
>> Thanks
>>
>> Stuti
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ::DISCLAIMER::
>>
>> ----------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> The contents of this e-mail and any attachment(s) are confidential and
>> intended for the named recipient(s) only.
>> E-mail transmission is not guaranteed to be secure or error-free as
>> information could be intercepted, corrupted,
>> lost, destroyed, arrive late or incomplete, or may contain viruses in
>> transmission. The e mail and its contents
>> (with or without referred errors) shall therefore not attach any
>> liability on the originator or HCL or its affiliates.
>> Views or opinions, if any, presented in this email are solely those of
>> the author and may not necessarily reflect the
>> views or opinions of HCL or its affiliates. Any form of reproduction,
>> dissemination, copying, disclosure, modification,
>> distribution and / or publication of this message without the prior
>> written consent of authorized representative of
>> HCL is strictly prohibited. If you have received this email in error
>> please delete it and notify the sender immediately.
>> Before opening any email and/or attachments, please check them for
>> viruses and other defects.
>>
>>
>> ----------------------------------------------------------------------------------------------------------------------------------------------------
>>
>>
>>
>
>
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Stuti Awasthi <st...@hcl.com>.
Hadoop version is 1.0.4
In hdfs-default.html for 1.0.4 version we have following property:
dfs.http.address
dfs.secondary.http.address
dfs.namenode.http-address : I suppose this property is not valid for Hadoop 1.x
Please suggest
Thanks
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 7:26 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010<http://10.139.9.238:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560<tel:4302274560> (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010<http://10.139.9.233:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010<http://10.139.9.232:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010<http://10.139.9.236:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544<tel:8669548544> (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010<http://10.139.9.235:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880<tel:5092986880> (4.74 GB)
Non DFS Used: 8669454336<tel:8669454336> (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010<http://10.139.9.237:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010<http://10.139.9.234:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440<tel:12124221440> (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010<http://10.139.9.231:50010>
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com<ma...@gmail.com>]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<http://%3cHostname%3e:50070/dfshealth.jsp> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
........
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Posted by Jitendra Yadav <je...@gmail.com>.
Correcting typo error.
*<name>dfs.namenode.http-address</name>*
*Thanks*
On Fri, Jan 31, 2014 at 7:25 PM, Jitendra Yadav
<je...@gmail.com>wrote:
> Can you please change below property and restart your cluster again?
>
> FROM:
>
> <name>dfs.http.address</name>
>
>
> TO:
> <name>dfs.namenode.http-addres</name>
>
> Thanks
> Jitendra
>
>
> On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>wrote:
>
>> Hi Jitendra,
>>
>>
>>
>> I realized that some days back ,my cluster was down due to power failure
>> after which nn/current directory has : edits, edits.new file and now SNN is
>> not rolling these edits due to HTTP error.
>>
>> Also currently my NN and SNN are operating on same machine
>>
>>
>>
>>
>>
>> DFSadmin report :
>>
>>
>>
>> Configured Capacity: 659494076416 (614.2 GB)
>>
>> Present Capacity: 535599210496 (498.82 GB)
>>
>> DFS Remaining: 497454006272 (463.29 GB)
>>
>> DFS Used: 38145204224 (35.53 GB)
>>
>> DFS Used%: 7.12%
>>
>> Under replicated blocks: 283
>>
>> Blocks with corrupt replicas: 3
>>
>> Missing blocks: 3
>>
>>
>>
>> -------------------------------------------------
>>
>> Datanodes available: 8 (8 total, 0 dead)
>>
>>
>>
>> Name: 10.139.9.238:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 4302274560 (4.01 GB)
>>
>> Non DFS Used: 8391843840 (7.82 GB)
>>
>> DFS Remaining: 69742641152(64.95 GB)
>>
>> DFS Used%: 5.22%
>>
>> DFS Remaining%: 84.6%
>>
>> Last contact: Fri Jan 31 18:55:18 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.233:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 5774745600 (5.38 GB)
>>
>> Non DFS Used: 13409488896 (12.49 GB)
>>
>> DFS Remaining: 63252525056(58.91 GB)
>>
>> DFS Used%: 7.01%
>>
>> DFS Remaining%: 76.73%
>>
>> Last contact: Fri Jan 31 18:55:19 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.232:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 8524451840 (7.94 GB)
>>
>> Non DFS Used: 24847884288 (23.14 GB)
>>
>> DFS Remaining: 49064423424(45.69 GB)
>>
>> DFS Used%: 10.34%
>>
>> DFS Remaining%: 59.52%
>>
>> Last contact: Fri Jan 31 18:55:21 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.236:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 4543819776 (4.23 GB)
>>
>> Non DFS Used: 8669548544 (8.07 GB)
>>
>> DFS Remaining: 69223391232(64.47 GB)
>>
>> DFS Used%: 5.51%
>>
>> DFS Remaining%: 83.97%
>>
>> Last contact: Fri Jan 31 18:55:19 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.235:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 5092986880 (4.74 GB)
>>
>> Non DFS Used: 8669454336 (8.07 GB)
>>
>> DFS Remaining: 68674318336(63.96 GB)
>>
>> DFS Used%: 6.18%
>>
>> DFS Remaining%: 83.31%
>>
>> Last contact: Fri Jan 31 18:55:19 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.237:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 4604301312 (4.29 GB)
>>
>> Non DFS Used: 11005788160 (10.25 GB)
>>
>> DFS Remaining: 66826670080(62.24 GB)
>>
>> DFS Used%: 5.59%
>>
>> DFS Remaining%: 81.06%
>>
>> Last contact: Fri Jan 31 18:55:18 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.234:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 4277760000 (3.98 GB)
>>
>> Non DFS Used: 12124221440 (11.29 GB)
>>
>> DFS Remaining: 66034778112(61.5 GB)
>>
>> DFS Used%: 5.19%
>>
>> DFS Remaining%: 80.1%
>>
>> Last contact: Fri Jan 31 18:55:18 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.231:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 1024864256 (977.39 MB)
>>
>> Non DFS Used: 36776636416 (34.25 GB)
>>
>> DFS Remaining: 44635258880(41.57 GB)
>>
>> DFS Used%: 1.24%
>>
>> DFS Remaining%: 54.14%
>>
>> Last contact: Fri Jan 31 18:55:20 IST 2014
>>
>>
>>
>>
>>
>>
>>
>> *From:* Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
>> *Sent:* Friday, January 31, 2014 6:58 PM
>> *To:* user
>> *Subject:* Re: java.io.FileNotFoundException:
>> http://HOSTNAME:50070/getimage?getimage=1
>>
>>
>>
>> Hi,
>>
>>
>>
>> Please post the output of dfs report command, this could help us to
>> understand cluster health.
>>
>>
>>
>> # *hadoop dfsadmin -report*
>>
>>
>>
>> Thanks
>>
>> Jitendra
>>
>>
>>
>> On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>
>> wrote:
>>
>> Hi All,
>>
>>
>>
>> I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP
>> request at port 50070 on dfs is not working properly.
>>
>> I have an Hadoop cluster which is operating from several days. Recently
>> we are also not able to see dfshealth.jsp page from webconsole.
>>
>>
>>
>> Problems :
>>
>> 1. http://<Hostname>:50070/dfshealth.jsp shows following error
>>
>>
>>
>> HTTP ERROR: 404
>>
>> Problem accessing /. Reason:
>>
>> NOT_FOUND
>>
>>
>>
>> 2. SNN is not able to roll edits :
>>
>> ERROR in SecondaryNameNode Log
>>
>> java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
>>
>> at
>> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
>>
>> at java.security.AccessController.doPrivileged(Native Method)
>>
>> at javax.security.auth.Subject.doAs(Subject.java:416)
>>
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
>>
>>
>>
>> ERROR in Namenode Log
>>
>> 2014-01-31 18:15:12,046 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>> 10.139.9.231
>>
>> 2014-01-31 18:15:12,046 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log,
>> edits.new files already exists in all healthy directories:
>>
>> /usr/lib/hadoop/storage/dfs/nn/current/edits.new
>>
>>
>>
>>
>>
>>
>>
>> Namenode logs which suggest that webserver is started on 50070
>> successfully:
>>
>> 2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port
>> returned by webServer.getConnectors()[0].getLocalPort() before open() is
>> -1. Opening the listener on 50070
>>
>> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer:
>> listener.getLocalPort() returned 50070
>> webServer.getConnectors()[0].getLocalPort() returned 50070
>>
>> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty
>> bound to port 50070
>>
>> 2014-01-31 14:42:35,378 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
>> HOSTNAME:50070
>>
>>
>>
>>
>>
>> Hdfs-site.xml
>>
>> <configuration>
>>
>> <property>
>>
>> <name>dfs.replication</name>
>>
>> <value>2</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>dfs.name.dir</name>
>>
>> <value>/usr/lib/hadoop/storage/dfs/nn</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>dfs.data.dir</name>
>>
>> <value>/usr/lib/hadoop/storage/dfs/dn</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>dfs.permissions</name>
>>
>> <value>false</value>
>>
>> </property>
>>
>> <property>
>>
>> <name>dfs.webhdfs.enabled</name>
>>
>> <value>true</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>dfs.http.address</name>
>>
>> <value>HOSTNAME:50070</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>dfs.secondary.http.address</name>
>>
>> <value>HOSTNAME:50090</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>fs.checkpoint.dir</name>
>>
>> <value>/usr/lib/hadoop/storage/dfs/snn</value>
>>
>> </property>
>>
>>
>>
>> </configuration>
>>
>>
>>
>>
>>
>> */etc/hosts* (Note I have also tried by commenting 127.0.0.1 entry in
>> host file but the issue was not resolved)
>>
>>
>>
>> 127.0.0.1 localhost
>>
>>
>>
>> IP1 Hostname1 # Namenode- vm01 - itself
>>
>> IP2 Hostname2 # DataNode- vm02
>>
>> ........
>>
>>
>>
>> # The following lines are desirable for IPv6 capable hosts
>>
>> ::1 ip6-localhost ip6-loopback
>>
>> fe00::0 ip6-localnet
>>
>> ff00::0 ip6-mcastprefix
>>
>> ff02::1 ip6-allnodes
>>
>> ff02::2 ip6-allrouters
>>
>>
>>
>>
>>
>> Note : All Hadoop daemons are executing fine and the jobs are running
>> properly.
>>
>>
>>
>> How to resolve this issue, I have tried many options provided on
>> different forums but still facing the same issue.
>>
>> I belive that this can cause a major problem later as my edits are not
>> getting rolled into fsimage.. This can cause me a data loss in case of
>> failure.
>>
>>
>>
>> Please suggest
>>
>>
>>
>> Thanks
>>
>> Stuti
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ::DISCLAIMER::
>>
>> ----------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> The contents of this e-mail and any attachment(s) are confidential and
>> intended for the named recipient(s) only.
>> E-mail transmission is not guaranteed to be secure or error-free as
>> information could be intercepted, corrupted,
>> lost, destroyed, arrive late or incomplete, or may contain viruses in
>> transmission. The e mail and its contents
>> (with or without referred errors) shall therefore not attach any
>> liability on the originator or HCL or its affiliates.
>> Views or opinions, if any, presented in this email are solely those of
>> the author and may not necessarily reflect the
>> views or opinions of HCL or its affiliates. Any form of reproduction,
>> dissemination, copying, disclosure, modification,
>> distribution and / or publication of this message without the prior
>> written consent of authorized representative of
>> HCL is strictly prohibited. If you have received this email in error
>> please delete it and notify the sender immediately.
>> Before opening any email and/or attachments, please check them for
>> viruses and other defects.
>>
>>
>> ----------------------------------------------------------------------------------------------------------------------------------------------------
>>
>>
>>
>
>
Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Posted by Jitendra Yadav <je...@gmail.com>.
Correcting typo error.
*<name>dfs.namenode.http-address</name>*
*Thanks*
On Fri, Jan 31, 2014 at 7:25 PM, Jitendra Yadav
<je...@gmail.com>wrote:
> Can you please change below property and restart your cluster again?
>
> FROM:
>
> <name>dfs.http.address</name>
>
>
> TO:
> <name>dfs.namenode.http-addres</name>
>
> Thanks
> Jitendra
>
>
> On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com>wrote:
>
>> Hi Jitendra,
>>
>>
>>
>> I realized that some days back ,my cluster was down due to power failure
>> after which nn/current directory has : edits, edits.new file and now SNN is
>> not rolling these edits due to HTTP error.
>>
>> Also currently my NN and SNN are operating on same machine
>>
>>
>>
>>
>>
>> DFSadmin report :
>>
>>
>>
>> Configured Capacity: 659494076416 (614.2 GB)
>>
>> Present Capacity: 535599210496 (498.82 GB)
>>
>> DFS Remaining: 497454006272 (463.29 GB)
>>
>> DFS Used: 38145204224 (35.53 GB)
>>
>> DFS Used%: 7.12%
>>
>> Under replicated blocks: 283
>>
>> Blocks with corrupt replicas: 3
>>
>> Missing blocks: 3
>>
>>
>>
>> -------------------------------------------------
>>
>> Datanodes available: 8 (8 total, 0 dead)
>>
>>
>>
>> Name: 10.139.9.238:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 4302274560 (4.01 GB)
>>
>> Non DFS Used: 8391843840 (7.82 GB)
>>
>> DFS Remaining: 69742641152(64.95 GB)
>>
>> DFS Used%: 5.22%
>>
>> DFS Remaining%: 84.6%
>>
>> Last contact: Fri Jan 31 18:55:18 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.233:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 5774745600 (5.38 GB)
>>
>> Non DFS Used: 13409488896 (12.49 GB)
>>
>> DFS Remaining: 63252525056(58.91 GB)
>>
>> DFS Used%: 7.01%
>>
>> DFS Remaining%: 76.73%
>>
>> Last contact: Fri Jan 31 18:55:19 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.232:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 8524451840 (7.94 GB)
>>
>> Non DFS Used: 24847884288 (23.14 GB)
>>
>> DFS Remaining: 49064423424(45.69 GB)
>>
>> DFS Used%: 10.34%
>>
>> DFS Remaining%: 59.52%
>>
>> Last contact: Fri Jan 31 18:55:21 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.236:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 4543819776 (4.23 GB)
>>
>> Non DFS Used: 8669548544 (8.07 GB)
>>
>> DFS Remaining: 69223391232(64.47 GB)
>>
>> DFS Used%: 5.51%
>>
>> DFS Remaining%: 83.97%
>>
>> Last contact: Fri Jan 31 18:55:19 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.235:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 5092986880 (4.74 GB)
>>
>> Non DFS Used: 8669454336 (8.07 GB)
>>
>> DFS Remaining: 68674318336(63.96 GB)
>>
>> DFS Used%: 6.18%
>>
>> DFS Remaining%: 83.31%
>>
>> Last contact: Fri Jan 31 18:55:19 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.237:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 4604301312 (4.29 GB)
>>
>> Non DFS Used: 11005788160 (10.25 GB)
>>
>> DFS Remaining: 66826670080(62.24 GB)
>>
>> DFS Used%: 5.59%
>>
>> DFS Remaining%: 81.06%
>>
>> Last contact: Fri Jan 31 18:55:18 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.234:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 4277760000 (3.98 GB)
>>
>> Non DFS Used: 12124221440 (11.29 GB)
>>
>> DFS Remaining: 66034778112(61.5 GB)
>>
>> DFS Used%: 5.19%
>>
>> DFS Remaining%: 80.1%
>>
>> Last contact: Fri Jan 31 18:55:18 IST 2014
>>
>>
>>
>>
>>
>> Name: 10.139.9.231:50010
>>
>> Decommission Status : Normal
>>
>> Configured Capacity: 82436759552 (76.78 GB)
>>
>> DFS Used: 1024864256 (977.39 MB)
>>
>> Non DFS Used: 36776636416 (34.25 GB)
>>
>> DFS Remaining: 44635258880(41.57 GB)
>>
>> DFS Used%: 1.24%
>>
>> DFS Remaining%: 54.14%
>>
>> Last contact: Fri Jan 31 18:55:20 IST 2014
>>
>>
>>
>>
>>
>>
>>
>> *From:* Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
>> *Sent:* Friday, January 31, 2014 6:58 PM
>> *To:* user
>> *Subject:* Re: java.io.FileNotFoundException:
>> http://HOSTNAME:50070/getimage?getimage=1
>>
>>
>>
>> Hi,
>>
>>
>>
>> Please post the output of dfs report command, this could help us to
>> understand cluster health.
>>
>>
>>
>> # *hadoop dfsadmin -report*
>>
>>
>>
>> Thanks
>>
>> Jitendra
>>
>>
>>
>> On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>
>> wrote:
>>
>> Hi All,
>>
>>
>>
>> I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP
>> request at port 50070 on dfs is not working properly.
>>
>> I have an Hadoop cluster which is operating from several days. Recently
>> we are also not able to see dfshealth.jsp page from webconsole.
>>
>>
>>
>> Problems :
>>
>> 1. http://<Hostname>:50070/dfshealth.jsp shows following error
>>
>>
>>
>> HTTP ERROR: 404
>>
>> Problem accessing /. Reason:
>>
>> NOT_FOUND
>>
>>
>>
>> 2. SNN is not able to roll edits :
>>
>> ERROR in SecondaryNameNode Log
>>
>> java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
>>
>> at
>> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
>>
>> at java.security.AccessController.doPrivileged(Native Method)
>>
>> at javax.security.auth.Subject.doAs(Subject.java:416)
>>
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
>>
>> at
>> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
>>
>>
>>
>> ERROR in Namenode Log
>>
>> 2014-01-31 18:15:12,046 INFO
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
>> 10.139.9.231
>>
>> 2014-01-31 18:15:12,046 WARN
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log,
>> edits.new files already exists in all healthy directories:
>>
>> /usr/lib/hadoop/storage/dfs/nn/current/edits.new
>>
>>
>>
>>
>>
>>
>>
>> Namenode logs which suggest that webserver is started on 50070
>> successfully:
>>
>> 2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port
>> returned by webServer.getConnectors()[0].getLocalPort() before open() is
>> -1. Opening the listener on 50070
>>
>> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer:
>> listener.getLocalPort() returned 50070
>> webServer.getConnectors()[0].getLocalPort() returned 50070
>>
>> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty
>> bound to port 50070
>>
>> 2014-01-31 14:42:35,378 INFO
>> org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
>> HOSTNAME:50070
>>
>>
>>
>>
>>
>> Hdfs-site.xml
>>
>> <configuration>
>>
>> <property>
>>
>> <name>dfs.replication</name>
>>
>> <value>2</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>dfs.name.dir</name>
>>
>> <value>/usr/lib/hadoop/storage/dfs/nn</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>dfs.data.dir</name>
>>
>> <value>/usr/lib/hadoop/storage/dfs/dn</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>dfs.permissions</name>
>>
>> <value>false</value>
>>
>> </property>
>>
>> <property>
>>
>> <name>dfs.webhdfs.enabled</name>
>>
>> <value>true</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>dfs.http.address</name>
>>
>> <value>HOSTNAME:50070</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>dfs.secondary.http.address</name>
>>
>> <value>HOSTNAME:50090</value>
>>
>> </property>
>>
>>
>>
>> <property>
>>
>> <name>fs.checkpoint.dir</name>
>>
>> <value>/usr/lib/hadoop/storage/dfs/snn</value>
>>
>> </property>
>>
>>
>>
>> </configuration>
>>
>>
>>
>>
>>
>> */etc/hosts* (Note I have also tried by commenting 127.0.0.1 entry in
>> host file but the issue was not resolved)
>>
>>
>>
>> 127.0.0.1 localhost
>>
>>
>>
>> IP1 Hostname1 # Namenode- vm01 - itself
>>
>> IP2 Hostname2 # DataNode- vm02
>>
>> ........
>>
>>
>>
>> # The following lines are desirable for IPv6 capable hosts
>>
>> ::1 ip6-localhost ip6-loopback
>>
>> fe00::0 ip6-localnet
>>
>> ff00::0 ip6-mcastprefix
>>
>> ff02::1 ip6-allnodes
>>
>> ff02::2 ip6-allrouters
>>
>>
>>
>>
>>
>> Note : All Hadoop daemons are executing fine and the jobs are running
>> properly.
>>
>>
>>
>> How to resolve this issue, I have tried many options provided on
>> different forums but still facing the same issue.
>>
>> I belive that this can cause a major problem later as my edits are not
>> getting rolled into fsimage.. This can cause me a data loss in case of
>> failure.
>>
>>
>>
>> Please suggest
>>
>>
>>
>> Thanks
>>
>> Stuti
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ::DISCLAIMER::
>>
>> ----------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> The contents of this e-mail and any attachment(s) are confidential and
>> intended for the named recipient(s) only.
>> E-mail transmission is not guaranteed to be secure or error-free as
>> information could be intercepted, corrupted,
>> lost, destroyed, arrive late or incomplete, or may contain viruses in
>> transmission. The e mail and its contents
>> (with or without referred errors) shall therefore not attach any
>> liability on the originator or HCL or its affiliates.
>> Views or opinions, if any, presented in this email are solely those of
>> the author and may not necessarily reflect the
>> views or opinions of HCL or its affiliates. Any form of reproduction,
>> dissemination, copying, disclosure, modification,
>> distribution and / or publication of this message without the prior
>> written consent of authorized representative of
>> HCL is strictly prohibited. If you have received this email in error
>> please delete it and notify the sender immediately.
>> Before opening any email and/or attachments, please check them for
>> viruses and other defects.
>>
>>
>> ----------------------------------------------------------------------------------------------------------------------------------------------------
>>
>>
>>
>
>
Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Posted by Jitendra Yadav <je...@gmail.com>.
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com> wrote:
> Hi Jitendra,
>
>
>
> I realized that some days back ,my cluster was down due to power failure
> after which nn/current directory has : edits, edits.new file and now SNN is
> not rolling these edits due to HTTP error.
>
> Also currently my NN and SNN are operating on same machine
>
>
>
>
>
> DFSadmin report :
>
>
>
> Configured Capacity: 659494076416 (614.2 GB)
>
> Present Capacity: 535599210496 (498.82 GB)
>
> DFS Remaining: 497454006272 (463.29 GB)
>
> DFS Used: 38145204224 (35.53 GB)
>
> DFS Used%: 7.12%
>
> Under replicated blocks: 283
>
> Blocks with corrupt replicas: 3
>
> Missing blocks: 3
>
>
>
> -------------------------------------------------
>
> Datanodes available: 8 (8 total, 0 dead)
>
>
>
> Name: 10.139.9.238:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4302274560 (4.01 GB)
>
> Non DFS Used: 8391843840 (7.82 GB)
>
> DFS Remaining: 69742641152(64.95 GB)
>
> DFS Used%: 5.22%
>
> DFS Remaining%: 84.6%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.233:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 5774745600 (5.38 GB)
>
> Non DFS Used: 13409488896 (12.49 GB)
>
> DFS Remaining: 63252525056(58.91 GB)
>
> DFS Used%: 7.01%
>
> DFS Remaining%: 76.73%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.232:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 8524451840 (7.94 GB)
>
> Non DFS Used: 24847884288 (23.14 GB)
>
> DFS Remaining: 49064423424(45.69 GB)
>
> DFS Used%: 10.34%
>
> DFS Remaining%: 59.52%
>
> Last contact: Fri Jan 31 18:55:21 IST 2014
>
>
>
>
>
> Name: 10.139.9.236:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4543819776 (4.23 GB)
>
> Non DFS Used: 8669548544 (8.07 GB)
>
> DFS Remaining: 69223391232(64.47 GB)
>
> DFS Used%: 5.51%
>
> DFS Remaining%: 83.97%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.235:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 5092986880 (4.74 GB)
>
> Non DFS Used: 8669454336 (8.07 GB)
>
> DFS Remaining: 68674318336(63.96 GB)
>
> DFS Used%: 6.18%
>
> DFS Remaining%: 83.31%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.237:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4604301312 (4.29 GB)
>
> Non DFS Used: 11005788160 (10.25 GB)
>
> DFS Remaining: 66826670080(62.24 GB)
>
> DFS Used%: 5.59%
>
> DFS Remaining%: 81.06%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.234:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4277760000 (3.98 GB)
>
> Non DFS Used: 12124221440 (11.29 GB)
>
> DFS Remaining: 66034778112(61.5 GB)
>
> DFS Used%: 5.19%
>
> DFS Remaining%: 80.1%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.231:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 1024864256 (977.39 MB)
>
> Non DFS Used: 36776636416 (34.25 GB)
>
> DFS Remaining: 44635258880(41.57 GB)
>
> DFS Used%: 1.24%
>
> DFS Remaining%: 54.14%
>
> Last contact: Fri Jan 31 18:55:20 IST 2014
>
>
>
>
>
>
>
> *From:* Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
> *Sent:* Friday, January 31, 2014 6:58 PM
> *To:* user
> *Subject:* Re: java.io.FileNotFoundException:
> http://HOSTNAME:50070/getimage?getimage=1
>
>
>
> Hi,
>
>
>
> Please post the output of dfs report command, this could help us to
> understand cluster health.
>
>
>
> # *hadoop dfsadmin -report*
>
>
>
> Thanks
>
> Jitendra
>
>
>
> On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>
> wrote:
>
> Hi All,
>
>
>
> I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP
> request at port 50070 on dfs is not working properly.
>
> I have an Hadoop cluster which is operating from several days. Recently we
> are also not able to see dfshealth.jsp page from webconsole.
>
>
>
> Problems :
>
> 1. http://<Hostname>:50070/dfshealth.jsp shows following error
>
>
>
> HTTP ERROR: 404
>
> Problem accessing /. Reason:
>
> NOT_FOUND
>
>
>
> 2. SNN is not able to roll edits :
>
> ERROR in SecondaryNameNode Log
>
> java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
>
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
>
> at
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:416)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
>
>
>
> ERROR in Namenode Log
>
> 2014-01-31 18:15:12,046 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 10.139.9.231
>
> 2014-01-31 18:15:12,046 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log,
> edits.new files already exists in all healthy directories:
>
> /usr/lib/hadoop/storage/dfs/nn/current/edits.new
>
>
>
>
>
>
>
> Namenode logs which suggest that webserver is started on 50070
> successfully:
>
> 2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1. Opening the listener on 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50070
> webServer.getConnectors()[0].getLocalPort() returned 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50070
>
> 2014-01-31 14:42:35,378 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
> HOSTNAME:50070
>
>
>
>
>
> Hdfs-site.xml
>
> <configuration>
>
> <property>
>
> <name>dfs.replication</name>
>
> <value>2</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.name.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/nn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.data.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/dn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.permissions</name>
>
> <value>false</value>
>
> </property>
>
> <property>
>
> <name>dfs.webhdfs.enabled</name>
>
> <value>true</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.http.address</name>
>
> <value>HOSTNAME:50070</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.secondary.http.address</name>
>
> <value>HOSTNAME:50090</value>
>
> </property>
>
>
>
> <property>
>
> <name>fs.checkpoint.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/snn</value>
>
> </property>
>
>
>
> </configuration>
>
>
>
>
>
> */etc/hosts* (Note I have also tried by commenting 127.0.0.1 entry in
> host file but the issue was not resolved)
>
>
>
> 127.0.0.1 localhost
>
>
>
> IP1 Hostname1 # Namenode- vm01 - itself
>
> IP2 Hostname2 # DataNode- vm02
>
> ........
>
>
>
> # The following lines are desirable for IPv6 capable hosts
>
> ::1 ip6-localhost ip6-loopback
>
> fe00::0 ip6-localnet
>
> ff00::0 ip6-mcastprefix
>
> ff02::1 ip6-allnodes
>
> ff02::2 ip6-allrouters
>
>
>
>
>
> Note : All Hadoop daemons are executing fine and the jobs are running
> properly.
>
>
>
> How to resolve this issue, I have tried many options provided on different
> forums but still facing the same issue.
>
> I belive that this can cause a major problem later as my edits are not
> getting rolled into fsimage.. This can cause me a data loss in case of
> failure.
>
>
>
> Please suggest
>
>
>
> Thanks
>
> Stuti
>
>
>
>
>
>
>
>
>
> ::DISCLAIMER::
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> E-mail transmission is not guaranteed to be secure or error-free as
> information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents
> (with or without referred errors) shall therefore not attach any liability
> on the originator or HCL or its affiliates.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the
> views or opinions of HCL or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification,
> distribution and / or publication of this message without the prior
> written consent of authorized representative of
> HCL is strictly prohibited. If you have received this email in error
> please delete it and notify the sender immediately.
> Before opening any email and/or attachments, please check them for viruses
> and other defects.
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
>
>
Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Posted by Jitendra Yadav <je...@gmail.com>.
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com> wrote:
> Hi Jitendra,
>
>
>
> I realized that some days back ,my cluster was down due to power failure
> after which nn/current directory has : edits, edits.new file and now SNN is
> not rolling these edits due to HTTP error.
>
> Also currently my NN and SNN are operating on same machine
>
>
>
>
>
> DFSadmin report :
>
>
>
> Configured Capacity: 659494076416 (614.2 GB)
>
> Present Capacity: 535599210496 (498.82 GB)
>
> DFS Remaining: 497454006272 (463.29 GB)
>
> DFS Used: 38145204224 (35.53 GB)
>
> DFS Used%: 7.12%
>
> Under replicated blocks: 283
>
> Blocks with corrupt replicas: 3
>
> Missing blocks: 3
>
>
>
> -------------------------------------------------
>
> Datanodes available: 8 (8 total, 0 dead)
>
>
>
> Name: 10.139.9.238:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4302274560 (4.01 GB)
>
> Non DFS Used: 8391843840 (7.82 GB)
>
> DFS Remaining: 69742641152(64.95 GB)
>
> DFS Used%: 5.22%
>
> DFS Remaining%: 84.6%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.233:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 5774745600 (5.38 GB)
>
> Non DFS Used: 13409488896 (12.49 GB)
>
> DFS Remaining: 63252525056(58.91 GB)
>
> DFS Used%: 7.01%
>
> DFS Remaining%: 76.73%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.232:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 8524451840 (7.94 GB)
>
> Non DFS Used: 24847884288 (23.14 GB)
>
> DFS Remaining: 49064423424(45.69 GB)
>
> DFS Used%: 10.34%
>
> DFS Remaining%: 59.52%
>
> Last contact: Fri Jan 31 18:55:21 IST 2014
>
>
>
>
>
> Name: 10.139.9.236:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4543819776 (4.23 GB)
>
> Non DFS Used: 8669548544 (8.07 GB)
>
> DFS Remaining: 69223391232(64.47 GB)
>
> DFS Used%: 5.51%
>
> DFS Remaining%: 83.97%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.235:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 5092986880 (4.74 GB)
>
> Non DFS Used: 8669454336 (8.07 GB)
>
> DFS Remaining: 68674318336(63.96 GB)
>
> DFS Used%: 6.18%
>
> DFS Remaining%: 83.31%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.237:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4604301312 (4.29 GB)
>
> Non DFS Used: 11005788160 (10.25 GB)
>
> DFS Remaining: 66826670080(62.24 GB)
>
> DFS Used%: 5.59%
>
> DFS Remaining%: 81.06%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.234:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4277760000 (3.98 GB)
>
> Non DFS Used: 12124221440 (11.29 GB)
>
> DFS Remaining: 66034778112(61.5 GB)
>
> DFS Used%: 5.19%
>
> DFS Remaining%: 80.1%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.231:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 1024864256 (977.39 MB)
>
> Non DFS Used: 36776636416 (34.25 GB)
>
> DFS Remaining: 44635258880(41.57 GB)
>
> DFS Used%: 1.24%
>
> DFS Remaining%: 54.14%
>
> Last contact: Fri Jan 31 18:55:20 IST 2014
>
>
>
>
>
>
>
> *From:* Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
> *Sent:* Friday, January 31, 2014 6:58 PM
> *To:* user
> *Subject:* Re: java.io.FileNotFoundException:
> http://HOSTNAME:50070/getimage?getimage=1
>
>
>
> Hi,
>
>
>
> Please post the output of dfs report command, this could help us to
> understand cluster health.
>
>
>
> # *hadoop dfsadmin -report*
>
>
>
> Thanks
>
> Jitendra
>
>
>
> On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>
> wrote:
>
> Hi All,
>
>
>
> I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP
> request at port 50070 on dfs is not working properly.
>
> I have an Hadoop cluster which is operating from several days. Recently we
> are also not able to see dfshealth.jsp page from webconsole.
>
>
>
> Problems :
>
> 1. http://<Hostname>:50070/dfshealth.jsp shows following error
>
>
>
> HTTP ERROR: 404
>
> Problem accessing /. Reason:
>
> NOT_FOUND
>
>
>
> 2. SNN is not able to roll edits :
>
> ERROR in SecondaryNameNode Log
>
> java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
>
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
>
> at
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:416)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
>
>
>
> ERROR in Namenode Log
>
> 2014-01-31 18:15:12,046 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 10.139.9.231
>
> 2014-01-31 18:15:12,046 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log,
> edits.new files already exists in all healthy directories:
>
> /usr/lib/hadoop/storage/dfs/nn/current/edits.new
>
>
>
>
>
>
>
> Namenode logs which suggest that webserver is started on 50070
> successfully:
>
> 2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1. Opening the listener on 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50070
> webServer.getConnectors()[0].getLocalPort() returned 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50070
>
> 2014-01-31 14:42:35,378 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
> HOSTNAME:50070
>
>
>
>
>
> Hdfs-site.xml
>
> <configuration>
>
> <property>
>
> <name>dfs.replication</name>
>
> <value>2</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.name.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/nn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.data.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/dn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.permissions</name>
>
> <value>false</value>
>
> </property>
>
> <property>
>
> <name>dfs.webhdfs.enabled</name>
>
> <value>true</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.http.address</name>
>
> <value>HOSTNAME:50070</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.secondary.http.address</name>
>
> <value>HOSTNAME:50090</value>
>
> </property>
>
>
>
> <property>
>
> <name>fs.checkpoint.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/snn</value>
>
> </property>
>
>
>
> </configuration>
>
>
>
>
>
> */etc/hosts* (Note I have also tried by commenting 127.0.0.1 entry in
> host file but the issue was not resolved)
>
>
>
> 127.0.0.1 localhost
>
>
>
> IP1 Hostname1 # Namenode- vm01 - itself
>
> IP2 Hostname2 # DataNode- vm02
>
> ........
>
>
>
> # The following lines are desirable for IPv6 capable hosts
>
> ::1 ip6-localhost ip6-loopback
>
> fe00::0 ip6-localnet
>
> ff00::0 ip6-mcastprefix
>
> ff02::1 ip6-allnodes
>
> ff02::2 ip6-allrouters
>
>
>
>
>
> Note : All Hadoop daemons are executing fine and the jobs are running
> properly.
>
>
>
> How to resolve this issue, I have tried many options provided on different
> forums but still facing the same issue.
>
> I belive that this can cause a major problem later as my edits are not
> getting rolled into fsimage.. This can cause me a data loss in case of
> failure.
>
>
>
> Please suggest
>
>
>
> Thanks
>
> Stuti
>
>
>
>
>
>
>
>
>
> ::DISCLAIMER::
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> E-mail transmission is not guaranteed to be secure or error-free as
> information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents
> (with or without referred errors) shall therefore not attach any liability
> on the originator or HCL or its affiliates.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the
> views or opinions of HCL or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification,
> distribution and / or publication of this message without the prior
> written consent of authorized representative of
> HCL is strictly prohibited. If you have received this email in error
> please delete it and notify the sender immediately.
> Before opening any email and/or attachments, please check them for viruses
> and other defects.
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
>
>
Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Posted by Jitendra Yadav <je...@gmail.com>.
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com> wrote:
> Hi Jitendra,
>
>
>
> I realized that some days back ,my cluster was down due to power failure
> after which nn/current directory has : edits, edits.new file and now SNN is
> not rolling these edits due to HTTP error.
>
> Also currently my NN and SNN are operating on same machine
>
>
>
>
>
> DFSadmin report :
>
>
>
> Configured Capacity: 659494076416 (614.2 GB)
>
> Present Capacity: 535599210496 (498.82 GB)
>
> DFS Remaining: 497454006272 (463.29 GB)
>
> DFS Used: 38145204224 (35.53 GB)
>
> DFS Used%: 7.12%
>
> Under replicated blocks: 283
>
> Blocks with corrupt replicas: 3
>
> Missing blocks: 3
>
>
>
> -------------------------------------------------
>
> Datanodes available: 8 (8 total, 0 dead)
>
>
>
> Name: 10.139.9.238:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4302274560 (4.01 GB)
>
> Non DFS Used: 8391843840 (7.82 GB)
>
> DFS Remaining: 69742641152(64.95 GB)
>
> DFS Used%: 5.22%
>
> DFS Remaining%: 84.6%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.233:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 5774745600 (5.38 GB)
>
> Non DFS Used: 13409488896 (12.49 GB)
>
> DFS Remaining: 63252525056(58.91 GB)
>
> DFS Used%: 7.01%
>
> DFS Remaining%: 76.73%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.232:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 8524451840 (7.94 GB)
>
> Non DFS Used: 24847884288 (23.14 GB)
>
> DFS Remaining: 49064423424(45.69 GB)
>
> DFS Used%: 10.34%
>
> DFS Remaining%: 59.52%
>
> Last contact: Fri Jan 31 18:55:21 IST 2014
>
>
>
>
>
> Name: 10.139.9.236:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4543819776 (4.23 GB)
>
> Non DFS Used: 8669548544 (8.07 GB)
>
> DFS Remaining: 69223391232(64.47 GB)
>
> DFS Used%: 5.51%
>
> DFS Remaining%: 83.97%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.235:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 5092986880 (4.74 GB)
>
> Non DFS Used: 8669454336 (8.07 GB)
>
> DFS Remaining: 68674318336(63.96 GB)
>
> DFS Used%: 6.18%
>
> DFS Remaining%: 83.31%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.237:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4604301312 (4.29 GB)
>
> Non DFS Used: 11005788160 (10.25 GB)
>
> DFS Remaining: 66826670080(62.24 GB)
>
> DFS Used%: 5.59%
>
> DFS Remaining%: 81.06%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.234:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4277760000 (3.98 GB)
>
> Non DFS Used: 12124221440 (11.29 GB)
>
> DFS Remaining: 66034778112(61.5 GB)
>
> DFS Used%: 5.19%
>
> DFS Remaining%: 80.1%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.231:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 1024864256 (977.39 MB)
>
> Non DFS Used: 36776636416 (34.25 GB)
>
> DFS Remaining: 44635258880(41.57 GB)
>
> DFS Used%: 1.24%
>
> DFS Remaining%: 54.14%
>
> Last contact: Fri Jan 31 18:55:20 IST 2014
>
>
>
>
>
>
>
> *From:* Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
> *Sent:* Friday, January 31, 2014 6:58 PM
> *To:* user
> *Subject:* Re: java.io.FileNotFoundException:
> http://HOSTNAME:50070/getimage?getimage=1
>
>
>
> Hi,
>
>
>
> Please post the output of dfs report command, this could help us to
> understand cluster health.
>
>
>
> # *hadoop dfsadmin -report*
>
>
>
> Thanks
>
> Jitendra
>
>
>
> On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>
> wrote:
>
> Hi All,
>
>
>
> I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP
> request at port 50070 on dfs is not working properly.
>
> I have an Hadoop cluster which is operating from several days. Recently we
> are also not able to see dfshealth.jsp page from webconsole.
>
>
>
> Problems :
>
> 1. http://<Hostname>:50070/dfshealth.jsp shows following error
>
>
>
> HTTP ERROR: 404
>
> Problem accessing /. Reason:
>
> NOT_FOUND
>
>
>
> 2. SNN is not able to roll edits :
>
> ERROR in SecondaryNameNode Log
>
> java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
>
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
>
> at
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:416)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
>
>
>
> ERROR in Namenode Log
>
> 2014-01-31 18:15:12,046 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 10.139.9.231
>
> 2014-01-31 18:15:12,046 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log,
> edits.new files already exists in all healthy directories:
>
> /usr/lib/hadoop/storage/dfs/nn/current/edits.new
>
>
>
>
>
>
>
> Namenode logs which suggest that webserver is started on 50070
> successfully:
>
> 2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1. Opening the listener on 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50070
> webServer.getConnectors()[0].getLocalPort() returned 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50070
>
> 2014-01-31 14:42:35,378 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
> HOSTNAME:50070
>
>
>
>
>
> Hdfs-site.xml
>
> <configuration>
>
> <property>
>
> <name>dfs.replication</name>
>
> <value>2</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.name.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/nn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.data.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/dn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.permissions</name>
>
> <value>false</value>
>
> </property>
>
> <property>
>
> <name>dfs.webhdfs.enabled</name>
>
> <value>true</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.http.address</name>
>
> <value>HOSTNAME:50070</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.secondary.http.address</name>
>
> <value>HOSTNAME:50090</value>
>
> </property>
>
>
>
> <property>
>
> <name>fs.checkpoint.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/snn</value>
>
> </property>
>
>
>
> </configuration>
>
>
>
>
>
> */etc/hosts* (Note I have also tried by commenting 127.0.0.1 entry in
> host file but the issue was not resolved)
>
>
>
> 127.0.0.1 localhost
>
>
>
> IP1 Hostname1 # Namenode- vm01 - itself
>
> IP2 Hostname2 # DataNode- vm02
>
> ........
>
>
>
> # The following lines are desirable for IPv6 capable hosts
>
> ::1 ip6-localhost ip6-loopback
>
> fe00::0 ip6-localnet
>
> ff00::0 ip6-mcastprefix
>
> ff02::1 ip6-allnodes
>
> ff02::2 ip6-allrouters
>
>
>
>
>
> Note : All Hadoop daemons are executing fine and the jobs are running
> properly.
>
>
>
> How to resolve this issue, I have tried many options provided on different
> forums but still facing the same issue.
>
> I belive that this can cause a major problem later as my edits are not
> getting rolled into fsimage.. This can cause me a data loss in case of
> failure.
>
>
>
> Please suggest
>
>
>
> Thanks
>
> Stuti
>
>
>
>
>
>
>
>
>
> ::DISCLAIMER::
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> E-mail transmission is not guaranteed to be secure or error-free as
> information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents
> (with or without referred errors) shall therefore not attach any liability
> on the originator or HCL or its affiliates.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the
> views or opinions of HCL or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification,
> distribution and / or publication of this message without the prior
> written consent of authorized representative of
> HCL is strictly prohibited. If you have received this email in error
> please delete it and notify the sender immediately.
> Before opening any email and/or attachments, please check them for viruses
> and other defects.
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
>
>
Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Posted by Jitendra Yadav <je...@gmail.com>.
Can you please change below property and restart your cluster again?
FROM:
<name>dfs.http.address</name>
TO:
<name>dfs.namenode.http-addres</name>
Thanks
Jitendra
On Fri, Jan 31, 2014 at 7:07 PM, Stuti Awasthi <st...@hcl.com> wrote:
> Hi Jitendra,
>
>
>
> I realized that some days back ,my cluster was down due to power failure
> after which nn/current directory has : edits, edits.new file and now SNN is
> not rolling these edits due to HTTP error.
>
> Also currently my NN and SNN are operating on same machine
>
>
>
>
>
> DFSadmin report :
>
>
>
> Configured Capacity: 659494076416 (614.2 GB)
>
> Present Capacity: 535599210496 (498.82 GB)
>
> DFS Remaining: 497454006272 (463.29 GB)
>
> DFS Used: 38145204224 (35.53 GB)
>
> DFS Used%: 7.12%
>
> Under replicated blocks: 283
>
> Blocks with corrupt replicas: 3
>
> Missing blocks: 3
>
>
>
> -------------------------------------------------
>
> Datanodes available: 8 (8 total, 0 dead)
>
>
>
> Name: 10.139.9.238:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4302274560 (4.01 GB)
>
> Non DFS Used: 8391843840 (7.82 GB)
>
> DFS Remaining: 69742641152(64.95 GB)
>
> DFS Used%: 5.22%
>
> DFS Remaining%: 84.6%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.233:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 5774745600 (5.38 GB)
>
> Non DFS Used: 13409488896 (12.49 GB)
>
> DFS Remaining: 63252525056(58.91 GB)
>
> DFS Used%: 7.01%
>
> DFS Remaining%: 76.73%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.232:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 8524451840 (7.94 GB)
>
> Non DFS Used: 24847884288 (23.14 GB)
>
> DFS Remaining: 49064423424(45.69 GB)
>
> DFS Used%: 10.34%
>
> DFS Remaining%: 59.52%
>
> Last contact: Fri Jan 31 18:55:21 IST 2014
>
>
>
>
>
> Name: 10.139.9.236:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4543819776 (4.23 GB)
>
> Non DFS Used: 8669548544 (8.07 GB)
>
> DFS Remaining: 69223391232(64.47 GB)
>
> DFS Used%: 5.51%
>
> DFS Remaining%: 83.97%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.235:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 5092986880 (4.74 GB)
>
> Non DFS Used: 8669454336 (8.07 GB)
>
> DFS Remaining: 68674318336(63.96 GB)
>
> DFS Used%: 6.18%
>
> DFS Remaining%: 83.31%
>
> Last contact: Fri Jan 31 18:55:19 IST 2014
>
>
>
>
>
> Name: 10.139.9.237:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4604301312 (4.29 GB)
>
> Non DFS Used: 11005788160 (10.25 GB)
>
> DFS Remaining: 66826670080(62.24 GB)
>
> DFS Used%: 5.59%
>
> DFS Remaining%: 81.06%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.234:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 4277760000 (3.98 GB)
>
> Non DFS Used: 12124221440 (11.29 GB)
>
> DFS Remaining: 66034778112(61.5 GB)
>
> DFS Used%: 5.19%
>
> DFS Remaining%: 80.1%
>
> Last contact: Fri Jan 31 18:55:18 IST 2014
>
>
>
>
>
> Name: 10.139.9.231:50010
>
> Decommission Status : Normal
>
> Configured Capacity: 82436759552 (76.78 GB)
>
> DFS Used: 1024864256 (977.39 MB)
>
> Non DFS Used: 36776636416 (34.25 GB)
>
> DFS Remaining: 44635258880(41.57 GB)
>
> DFS Used%: 1.24%
>
> DFS Remaining%: 54.14%
>
> Last contact: Fri Jan 31 18:55:20 IST 2014
>
>
>
>
>
>
>
> *From:* Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
> *Sent:* Friday, January 31, 2014 6:58 PM
> *To:* user
> *Subject:* Re: java.io.FileNotFoundException:
> http://HOSTNAME:50070/getimage?getimage=1
>
>
>
> Hi,
>
>
>
> Please post the output of dfs report command, this could help us to
> understand cluster health.
>
>
>
> # *hadoop dfsadmin -report*
>
>
>
> Thanks
>
> Jitendra
>
>
>
> On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>
> wrote:
>
> Hi All,
>
>
>
> I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP
> request at port 50070 on dfs is not working properly.
>
> I have an Hadoop cluster which is operating from several days. Recently we
> are also not able to see dfshealth.jsp page from webconsole.
>
>
>
> Problems :
>
> 1. http://<Hostname>:50070/dfshealth.jsp shows following error
>
>
>
> HTTP ERROR: 404
>
> Problem accessing /. Reason:
>
> NOT_FOUND
>
>
>
> 2. SNN is not able to roll edits :
>
> ERROR in SecondaryNameNode Log
>
> java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
>
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
>
> at
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:416)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
>
>
>
> ERROR in Namenode Log
>
> 2014-01-31 18:15:12,046 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 10.139.9.231
>
> 2014-01-31 18:15:12,046 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log,
> edits.new files already exists in all healthy directories:
>
> /usr/lib/hadoop/storage/dfs/nn/current/edits.new
>
>
>
>
>
>
>
> Namenode logs which suggest that webserver is started on 50070
> successfully:
>
> 2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1. Opening the listener on 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50070
> webServer.getConnectors()[0].getLocalPort() returned 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50070
>
> 2014-01-31 14:42:35,378 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
> HOSTNAME:50070
>
>
>
>
>
> Hdfs-site.xml
>
> <configuration>
>
> <property>
>
> <name>dfs.replication</name>
>
> <value>2</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.name.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/nn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.data.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/dn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.permissions</name>
>
> <value>false</value>
>
> </property>
>
> <property>
>
> <name>dfs.webhdfs.enabled</name>
>
> <value>true</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.http.address</name>
>
> <value>HOSTNAME:50070</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.secondary.http.address</name>
>
> <value>HOSTNAME:50090</value>
>
> </property>
>
>
>
> <property>
>
> <name>fs.checkpoint.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/snn</value>
>
> </property>
>
>
>
> </configuration>
>
>
>
>
>
> */etc/hosts* (Note I have also tried by commenting 127.0.0.1 entry in
> host file but the issue was not resolved)
>
>
>
> 127.0.0.1 localhost
>
>
>
> IP1 Hostname1 # Namenode- vm01 - itself
>
> IP2 Hostname2 # DataNode- vm02
>
> ........
>
>
>
> # The following lines are desirable for IPv6 capable hosts
>
> ::1 ip6-localhost ip6-loopback
>
> fe00::0 ip6-localnet
>
> ff00::0 ip6-mcastprefix
>
> ff02::1 ip6-allnodes
>
> ff02::2 ip6-allrouters
>
>
>
>
>
> Note : All Hadoop daemons are executing fine and the jobs are running
> properly.
>
>
>
> How to resolve this issue, I have tried many options provided on different
> forums but still facing the same issue.
>
> I belive that this can cause a major problem later as my edits are not
> getting rolled into fsimage.. This can cause me a data loss in case of
> failure.
>
>
>
> Please suggest
>
>
>
> Thanks
>
> Stuti
>
>
>
>
>
>
>
>
>
> ::DISCLAIMER::
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> E-mail transmission is not guaranteed to be secure or error-free as
> information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents
> (with or without referred errors) shall therefore not attach any liability
> on the originator or HCL or its affiliates.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the
> views or opinions of HCL or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification,
> distribution and / or publication of this message without the prior
> written consent of authorized representative of
> HCL is strictly prohibited. If you have received this email in error
> please delete it and notify the sender immediately.
> Before opening any email and/or attachments, please check them for viruses
> and other defects.
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
>
>
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Stuti Awasthi <st...@hcl.com>.
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560 (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544 (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880 (4.74 GB)
Non DFS Used: 8669454336 (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440 (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<http://%3cHostname%3e:50070/dfshealth.jsp> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
........
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Stuti Awasthi <st...@hcl.com>.
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560 (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544 (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880 (4.74 GB)
Non DFS Used: 8669454336 (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440 (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<http://%3cHostname%3e:50070/dfshealth.jsp> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
........
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Stuti Awasthi <st...@hcl.com>.
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560 (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544 (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880 (4.74 GB)
Non DFS Used: 8669454336 (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440 (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<http://%3cHostname%3e:50070/dfshealth.jsp> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
........
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
RE: java.io.FileNotFoundException:
http://HOSTNAME:50070/getimage?getimage=1
Posted by Stuti Awasthi <st...@hcl.com>.
Hi Jitendra,
I realized that some days back ,my cluster was down due to power failure after which nn/current directory has : edits, edits.new file and now SNN is not rolling these edits due to HTTP error.
Also currently my NN and SNN are operating on same machine
DFSadmin report :
Configured Capacity: 659494076416 (614.2 GB)
Present Capacity: 535599210496 (498.82 GB)
DFS Remaining: 497454006272 (463.29 GB)
DFS Used: 38145204224 (35.53 GB)
DFS Used%: 7.12%
Under replicated blocks: 283
Blocks with corrupt replicas: 3
Missing blocks: 3
-------------------------------------------------
Datanodes available: 8 (8 total, 0 dead)
Name: 10.139.9.238:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4302274560 (4.01 GB)
Non DFS Used: 8391843840 (7.82 GB)
DFS Remaining: 69742641152(64.95 GB)
DFS Used%: 5.22%
DFS Remaining%: 84.6%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.233:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5774745600 (5.38 GB)
Non DFS Used: 13409488896 (12.49 GB)
DFS Remaining: 63252525056(58.91 GB)
DFS Used%: 7.01%
DFS Remaining%: 76.73%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.232:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 8524451840 (7.94 GB)
Non DFS Used: 24847884288 (23.14 GB)
DFS Remaining: 49064423424(45.69 GB)
DFS Used%: 10.34%
DFS Remaining%: 59.52%
Last contact: Fri Jan 31 18:55:21 IST 2014
Name: 10.139.9.236:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4543819776 (4.23 GB)
Non DFS Used: 8669548544 (8.07 GB)
DFS Remaining: 69223391232(64.47 GB)
DFS Used%: 5.51%
DFS Remaining%: 83.97%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.235:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 5092986880 (4.74 GB)
Non DFS Used: 8669454336 (8.07 GB)
DFS Remaining: 68674318336(63.96 GB)
DFS Used%: 6.18%
DFS Remaining%: 83.31%
Last contact: Fri Jan 31 18:55:19 IST 2014
Name: 10.139.9.237:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4604301312 (4.29 GB)
Non DFS Used: 11005788160 (10.25 GB)
DFS Remaining: 66826670080(62.24 GB)
DFS Used%: 5.59%
DFS Remaining%: 81.06%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.234:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 4277760000 (3.98 GB)
Non DFS Used: 12124221440 (11.29 GB)
DFS Remaining: 66034778112(61.5 GB)
DFS Used%: 5.19%
DFS Remaining%: 80.1%
Last contact: Fri Jan 31 18:55:18 IST 2014
Name: 10.139.9.231:50010
Decommission Status : Normal
Configured Capacity: 82436759552 (76.78 GB)
DFS Used: 1024864256 (977.39 MB)
Non DFS Used: 36776636416 (34.25 GB)
DFS Remaining: 44635258880(41.57 GB)
DFS Used%: 1.24%
DFS Remaining%: 54.14%
Last contact: Fri Jan 31 18:55:20 IST 2014
From: Jitendra Yadav [mailto:jeetuyadav200890@gmail.com]
Sent: Friday, January 31, 2014 6:58 PM
To: user
Subject: Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Hi,
Please post the output of dfs report command, this could help us to understand cluster health.
# hadoop dfsadmin -report
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com>> wrote:
Hi All,
I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP request at port 50070 on dfs is not working properly.
I have an Hadoop cluster which is operating from several days. Recently we are also not able to see dfshealth.jsp page from webconsole.
Problems :
1. http://<Hostname>:50070/dfshealth.jsp<http://%3cHostname%3e:50070/dfshealth.jsp> shows following error
HTTP ERROR: 404
Problem accessing /. Reason:
NOT_FOUND
2. SNN is not able to roll edits :
ERROR in SecondaryNameNode Log
java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
at org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
ERROR in Namenode Log
2014-01-31 18:15:12,046 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 10.139.9.231
2014-01-31 18:15:12,046 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log, edits.new files already exists in all healthy directories:
/usr/lib/hadoop/storage/dfs/nn/current/edits.new
Namenode logs which suggest that webserver is started on 50070 successfully:
2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2014-01-31 14:42:35,378 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: HOSTNAME:50070
Hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/lib/hadoop/storage/dfs/nn</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/lib/hadoop/storage/dfs/dn</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.http.address</name>
<value>HOSTNAME:50070</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>HOSTNAME:50090</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/usr/lib/hadoop/storage/dfs/snn</value>
</property>
</configuration>
/etc/hosts (Note I have also tried by commenting 127.0.0.1 entry in host file but the issue was not resolved)
127.0.0.1 localhost
IP1 Hostname1 # Namenode- vm01 - itself
IP2 Hostname2 # DataNode- vm02
........
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Note : All Hadoop daemons are executing fine and the jobs are running properly.
How to resolve this issue, I have tried many options provided on different forums but still facing the same issue.
I belive that this can cause a major problem later as my edits are not getting rolled into fsimage.. This can cause me a data loss in case of failure.
Please suggest
Thanks
Stuti
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Posted by Jitendra Yadav <je...@gmail.com>.
Hi,
Please post the output of dfs report command, this could help us to
understand cluster health.
# *hadoop dfsadmin -report*
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com> wrote:
> Hi All,
>
>
>
> I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP
> request at port 50070 on dfs is not working properly.
>
> I have an Hadoop cluster which is operating from several days. Recently we
> are also not able to see dfshealth.jsp page from webconsole.
>
>
>
> Problems :
>
> 1. http://<Hostname>:50070/dfshealth.jsp shows following error
>
>
>
> HTTP ERROR: 404
>
> Problem accessing /. Reason:
>
> NOT_FOUND
>
>
>
> 2. SNN is not able to roll edits :
>
> ERROR in SecondaryNameNode Log
>
> java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
>
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
>
> at
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:416)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
>
>
>
> ERROR in Namenode Log
>
> 2014-01-31 18:15:12,046 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 10.139.9.231
>
> 2014-01-31 18:15:12,046 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log,
> edits.new files already exists in all healthy directories:
>
> /usr/lib/hadoop/storage/dfs/nn/current/edits.new
>
>
>
>
>
>
>
> Namenode logs which suggest that webserver is started on 50070
> successfully:
>
> 2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1. Opening the listener on 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50070
> webServer.getConnectors()[0].getLocalPort() returned 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50070
>
> 2014-01-31 14:42:35,378 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
> HOSTNAME:50070
>
>
>
>
>
> Hdfs-site.xml
>
> <configuration>
>
> <property>
>
> <name>dfs.replication</name>
>
> <value>2</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.name.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/nn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.data.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/dn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.permissions</name>
>
> <value>false</value>
>
> </property>
>
> <property>
>
> <name>dfs.webhdfs.enabled</name>
>
> <value>true</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.http.address</name>
>
> <value>HOSTNAME:50070</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.secondary.http.address</name>
>
> <value>HOSTNAME:50090</value>
>
> </property>
>
>
>
> <property>
>
> <name>fs.checkpoint.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/snn</value>
>
> </property>
>
>
>
> </configuration>
>
>
>
>
>
> */etc/hosts* (Note I have also tried by commenting 127.0.0.1 entry in
> host file but the issue was not resolved)
>
>
>
> 127.0.0.1 localhost
>
>
>
> IP1 Hostname1 # Namenode- vm01 - itself
>
> IP2 Hostname2 # DataNode- vm02
>
> ........
>
>
>
> # The following lines are desirable for IPv6 capable hosts
>
> ::1 ip6-localhost ip6-loopback
>
> fe00::0 ip6-localnet
>
> ff00::0 ip6-mcastprefix
>
> ff02::1 ip6-allnodes
>
> ff02::2 ip6-allrouters
>
>
>
>
>
> Note : All Hadoop daemons are executing fine and the jobs are running
> properly.
>
>
>
> How to resolve this issue, I have tried many options provided on different
> forums but still facing the same issue.
>
> I belive that this can cause a major problem later as my edits are not
> getting rolled into fsimage.. This can cause me a data loss in case of
> failure.
>
>
>
> Please suggest
>
>
>
> Thanks
>
> Stuti
>
>
>
>
>
>
>
>
>
> ::DISCLAIMER::
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> E-mail transmission is not guaranteed to be secure or error-free as
> information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents
> (with or without referred errors) shall therefore not attach any liability
> on the originator or HCL or its affiliates.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the
> views or opinions of HCL or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification,
> distribution and / or publication of this message without the prior
> written consent of authorized representative of
> HCL is strictly prohibited. If you have received this email in error
> please delete it and notify the sender immediately.
> Before opening any email and/or attachments, please check them for viruses
> and other defects.
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Posted by Jitendra Yadav <je...@gmail.com>.
Hi,
Please post the output of dfs report command, this could help us to
understand cluster health.
# *hadoop dfsadmin -report*
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com> wrote:
> Hi All,
>
>
>
> I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP
> request at port 50070 on dfs is not working properly.
>
> I have an Hadoop cluster which is operating from several days. Recently we
> are also not able to see dfshealth.jsp page from webconsole.
>
>
>
> Problems :
>
> 1. http://<Hostname>:50070/dfshealth.jsp shows following error
>
>
>
> HTTP ERROR: 404
>
> Problem accessing /. Reason:
>
> NOT_FOUND
>
>
>
> 2. SNN is not able to roll edits :
>
> ERROR in SecondaryNameNode Log
>
> java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
>
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
>
> at
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:416)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
>
>
>
> ERROR in Namenode Log
>
> 2014-01-31 18:15:12,046 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 10.139.9.231
>
> 2014-01-31 18:15:12,046 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log,
> edits.new files already exists in all healthy directories:
>
> /usr/lib/hadoop/storage/dfs/nn/current/edits.new
>
>
>
>
>
>
>
> Namenode logs which suggest that webserver is started on 50070
> successfully:
>
> 2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1. Opening the listener on 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50070
> webServer.getConnectors()[0].getLocalPort() returned 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50070
>
> 2014-01-31 14:42:35,378 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
> HOSTNAME:50070
>
>
>
>
>
> Hdfs-site.xml
>
> <configuration>
>
> <property>
>
> <name>dfs.replication</name>
>
> <value>2</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.name.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/nn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.data.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/dn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.permissions</name>
>
> <value>false</value>
>
> </property>
>
> <property>
>
> <name>dfs.webhdfs.enabled</name>
>
> <value>true</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.http.address</name>
>
> <value>HOSTNAME:50070</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.secondary.http.address</name>
>
> <value>HOSTNAME:50090</value>
>
> </property>
>
>
>
> <property>
>
> <name>fs.checkpoint.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/snn</value>
>
> </property>
>
>
>
> </configuration>
>
>
>
>
>
> */etc/hosts* (Note I have also tried by commenting 127.0.0.1 entry in
> host file but the issue was not resolved)
>
>
>
> 127.0.0.1 localhost
>
>
>
> IP1 Hostname1 # Namenode- vm01 - itself
>
> IP2 Hostname2 # DataNode- vm02
>
> ........
>
>
>
> # The following lines are desirable for IPv6 capable hosts
>
> ::1 ip6-localhost ip6-loopback
>
> fe00::0 ip6-localnet
>
> ff00::0 ip6-mcastprefix
>
> ff02::1 ip6-allnodes
>
> ff02::2 ip6-allrouters
>
>
>
>
>
> Note : All Hadoop daemons are executing fine and the jobs are running
> properly.
>
>
>
> How to resolve this issue, I have tried many options provided on different
> forums but still facing the same issue.
>
> I belive that this can cause a major problem later as my edits are not
> getting rolled into fsimage.. This can cause me a data loss in case of
> failure.
>
>
>
> Please suggest
>
>
>
> Thanks
>
> Stuti
>
>
>
>
>
>
>
>
>
> ::DISCLAIMER::
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> E-mail transmission is not guaranteed to be secure or error-free as
> information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents
> (with or without referred errors) shall therefore not attach any liability
> on the originator or HCL or its affiliates.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the
> views or opinions of HCL or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification,
> distribution and / or publication of this message without the prior
> written consent of authorized representative of
> HCL is strictly prohibited. If you have received this email in error
> please delete it and notify the sender immediately.
> Before opening any email and/or attachments, please check them for viruses
> and other defects.
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Posted by Jitendra Yadav <je...@gmail.com>.
Hi,
Please post the output of dfs report command, this could help us to
understand cluster health.
# *hadoop dfsadmin -report*
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com> wrote:
> Hi All,
>
>
>
> I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP
> request at port 50070 on dfs is not working properly.
>
> I have an Hadoop cluster which is operating from several days. Recently we
> are also not able to see dfshealth.jsp page from webconsole.
>
>
>
> Problems :
>
> 1. http://<Hostname>:50070/dfshealth.jsp shows following error
>
>
>
> HTTP ERROR: 404
>
> Problem accessing /. Reason:
>
> NOT_FOUND
>
>
>
> 2. SNN is not able to roll edits :
>
> ERROR in SecondaryNameNode Log
>
> java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
>
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
>
> at
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:416)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
>
>
>
> ERROR in Namenode Log
>
> 2014-01-31 18:15:12,046 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 10.139.9.231
>
> 2014-01-31 18:15:12,046 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log,
> edits.new files already exists in all healthy directories:
>
> /usr/lib/hadoop/storage/dfs/nn/current/edits.new
>
>
>
>
>
>
>
> Namenode logs which suggest that webserver is started on 50070
> successfully:
>
> 2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1. Opening the listener on 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50070
> webServer.getConnectors()[0].getLocalPort() returned 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50070
>
> 2014-01-31 14:42:35,378 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
> HOSTNAME:50070
>
>
>
>
>
> Hdfs-site.xml
>
> <configuration>
>
> <property>
>
> <name>dfs.replication</name>
>
> <value>2</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.name.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/nn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.data.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/dn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.permissions</name>
>
> <value>false</value>
>
> </property>
>
> <property>
>
> <name>dfs.webhdfs.enabled</name>
>
> <value>true</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.http.address</name>
>
> <value>HOSTNAME:50070</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.secondary.http.address</name>
>
> <value>HOSTNAME:50090</value>
>
> </property>
>
>
>
> <property>
>
> <name>fs.checkpoint.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/snn</value>
>
> </property>
>
>
>
> </configuration>
>
>
>
>
>
> */etc/hosts* (Note I have also tried by commenting 127.0.0.1 entry in
> host file but the issue was not resolved)
>
>
>
> 127.0.0.1 localhost
>
>
>
> IP1 Hostname1 # Namenode- vm01 - itself
>
> IP2 Hostname2 # DataNode- vm02
>
> ........
>
>
>
> # The following lines are desirable for IPv6 capable hosts
>
> ::1 ip6-localhost ip6-loopback
>
> fe00::0 ip6-localnet
>
> ff00::0 ip6-mcastprefix
>
> ff02::1 ip6-allnodes
>
> ff02::2 ip6-allrouters
>
>
>
>
>
> Note : All Hadoop daemons are executing fine and the jobs are running
> properly.
>
>
>
> How to resolve this issue, I have tried many options provided on different
> forums but still facing the same issue.
>
> I belive that this can cause a major problem later as my edits are not
> getting rolled into fsimage.. This can cause me a data loss in case of
> failure.
>
>
>
> Please suggest
>
>
>
> Thanks
>
> Stuti
>
>
>
>
>
>
>
>
>
> ::DISCLAIMER::
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> E-mail transmission is not guaranteed to be secure or error-free as
> information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents
> (with or without referred errors) shall therefore not attach any liability
> on the originator or HCL or its affiliates.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the
> views or opinions of HCL or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification,
> distribution and / or publication of this message without the prior
> written consent of authorized representative of
> HCL is strictly prohibited. If you have received this email in error
> please delete it and notify the sender immediately.
> Before opening any email and/or attachments, please check them for viruses
> and other defects.
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
Re: java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
Posted by Jitendra Yadav <je...@gmail.com>.
Hi,
Please post the output of dfs report command, this could help us to
understand cluster health.
# *hadoop dfsadmin -report*
Thanks
Jitendra
On Fri, Jan 31, 2014 at 6:44 PM, Stuti Awasthi <st...@hcl.com> wrote:
> Hi All,
>
>
>
> I am suddenly started facing issue on Hadoop Cluster. Seems like HTTP
> request at port 50070 on dfs is not working properly.
>
> I have an Hadoop cluster which is operating from several days. Recently we
> are also not able to see dfshealth.jsp page from webconsole.
>
>
>
> Problems :
>
> 1. http://<Hostname>:50070/dfshealth.jsp shows following error
>
>
>
> HTTP ERROR: 404
>
> Problem accessing /. Reason:
>
> NOT_FOUND
>
>
>
> 2. SNN is not able to roll edits :
>
> ERROR in SecondaryNameNode Log
>
> java.io.FileNotFoundException: http://HOSTNAME:50070/getimage?getimage=1
>
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)
>
> at
> org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:160)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:347)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$3.run(SecondaryNameNode.java:336)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:416)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.downloadCheckpointFiles(SecondaryNameNode.java:336)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:411)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:312)
>
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:275)
>
>
>
> ERROR in Namenode Log
>
> 2014-01-31 18:15:12,046 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> 10.139.9.231
>
> 2014-01-31 18:15:12,046 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit log,
> edits.new files already exists in all healthy directories:
>
> /usr/lib/hadoop/storage/dfs/nn/current/edits.new
>
>
>
>
>
>
>
> Namenode logs which suggest that webserver is started on 50070
> successfully:
>
> 2014-01-31 14:42:35,208 INFO org.apache.hadoop.http.HttpServer: Port
> returned by webServer.getConnectors()[0].getLocalPort() before open() is
> -1. Opening the listener on 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer:
> listener.getLocalPort() returned 50070
> webServer.getConnectors()[0].getLocalPort() returned 50070
>
> 2014-01-31 14:42:35,209 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound to port 50070
>
> 2014-01-31 14:42:35,378 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at:
> HOSTNAME:50070
>
>
>
>
>
> Hdfs-site.xml
>
> <configuration>
>
> <property>
>
> <name>dfs.replication</name>
>
> <value>2</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.name.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/nn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.data.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/dn</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.permissions</name>
>
> <value>false</value>
>
> </property>
>
> <property>
>
> <name>dfs.webhdfs.enabled</name>
>
> <value>true</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.http.address</name>
>
> <value>HOSTNAME:50070</value>
>
> </property>
>
>
>
> <property>
>
> <name>dfs.secondary.http.address</name>
>
> <value>HOSTNAME:50090</value>
>
> </property>
>
>
>
> <property>
>
> <name>fs.checkpoint.dir</name>
>
> <value>/usr/lib/hadoop/storage/dfs/snn</value>
>
> </property>
>
>
>
> </configuration>
>
>
>
>
>
> */etc/hosts* (Note I have also tried by commenting 127.0.0.1 entry in
> host file but the issue was not resolved)
>
>
>
> 127.0.0.1 localhost
>
>
>
> IP1 Hostname1 # Namenode- vm01 - itself
>
> IP2 Hostname2 # DataNode- vm02
>
> ........
>
>
>
> # The following lines are desirable for IPv6 capable hosts
>
> ::1 ip6-localhost ip6-loopback
>
> fe00::0 ip6-localnet
>
> ff00::0 ip6-mcastprefix
>
> ff02::1 ip6-allnodes
>
> ff02::2 ip6-allrouters
>
>
>
>
>
> Note : All Hadoop daemons are executing fine and the jobs are running
> properly.
>
>
>
> How to resolve this issue, I have tried many options provided on different
> forums but still facing the same issue.
>
> I belive that this can cause a major problem later as my edits are not
> getting rolled into fsimage.. This can cause me a data loss in case of
> failure.
>
>
>
> Please suggest
>
>
>
> Thanks
>
> Stuti
>
>
>
>
>
>
>
>
>
> ::DISCLAIMER::
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> E-mail transmission is not guaranteed to be secure or error-free as
> information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents
> (with or without referred errors) shall therefore not attach any liability
> on the originator or HCL or its affiliates.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the
> views or opinions of HCL or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification,
> distribution and / or publication of this message without the prior
> written consent of authorized representative of
> HCL is strictly prohibited. If you have received this email in error
> please delete it and notify the sender immediately.
> Before opening any email and/or attachments, please check them for viruses
> and other defects.
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>