You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Terry Healy <th...@bnl.gov> on 2012/08/15 18:39:13 UTC
Re: 'Cannot roll edit log'
This situation continues and reports every 10 minutes. I even tried
moving the secondary NN function to a different node. I have run TCPDump
but cannot isolate the "Connection Refused" issue. Am I correct in
assuming that the NN will try to connect to SNN on port 50090?
Any way out? Current partial dump below.
2012-08-15 12:36:09,422 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
xxx.yyy.254.254
2012-08-15 12:36:09,423 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit
log, edits.new files already exists in all healthy directories:
/home/thealy/hdfs/name/current/edits.new
2012-08-15 12:36:09,879 ERROR
org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:thealy cause:java.net.ConnectException:
Connection refused
2012-08-15 12:36:09,879 ERROR
org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:thealy cause:java.net.ConnectException:
Connection refused
2012-08-15 12:36:09,881 WARN org.mortbay.log: /getimage:
java.io.IOException: GetImage failed. java.net.ConnectException:
Connection refused
On 07/09/2012 01:23 PM, Terry Healy wrote:
> Any suggestions on how to clear the error? stp-all / start-all had no
> effect.
>
> On 07/03/2012 04:22 PM, Brandon Li wrote:
>> With 1.0.2, only one checkpoint process is executed at a time. When the
>> namenode gets an overlapping checkpointing request, it checks edit.new
>> in its storage directories. If all of them have this file, namenode
>> concludes the previous checkpoint process is not done yet and prints the
>> warning message you've seen.
>>
>> Brandon
>>
>> On Tue, Jul 3, 2012 at 10:56 AM, Terry Healy <thealy@bnl.gov
>> <ma...@bnl.gov>> wrote:
>>
>> Running Apache 1.0.2.
>>
>> The NN log is reporting that it cannot "roll the edit log" from the
>> secondary NN. The SecondaryNameNode is running on the system referred to
>> as xxx.yyy.254.238 in the log snippet below.
>>
>> From the NN, I can connect to the Secondary via ssh as the user. Any
>> suggestions what have I got wrong here?
>>
>> thanks,
>>
>> Terry
>>
>> INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>> from xxx.yyy.254.238
>> WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll
>> edit log, edits.new files already exists in all healthy directories:
>> /home/[user]/hdfs/name/current/edits.new
>>
>> ERROR org.apache.hadoop.security.UserGroupInformation:
>> PriviledgedActionException as:[user] cause:java.net.ConnectException:
>> Connection refused
>> ERROR org.apache.hadoop.security.UserGroupInformation:
>> PriviledgedActionException as:[user] cause:java.net.ConnectException:
>> Connection refused
>>
>> WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed.
>> java.net.ConnectException: Connection refused
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
>> ....
>>
>>
>> --
>> Terry Healy / thealy@bnl.gov <ma...@bnl.gov>
>> Cyber Security Operations
>> Brookhaven National Laboratory
>> Building 515, Upton N.Y. 11973
>>
>>
>
--
Terry Healy / thealy@bnl.gov
Cyber Security Operations
Brookhaven National Laboratory
Building 515, Upton N.Y. 11973
Re: 'Cannot roll edit log'
Posted by Terry Healy <th...@bnl.gov>.
Further info: From the log of secondary NN:
INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Posted
URL
AAA.xxx.yyy.zzz:50070putimage=1&port=50090&machine=0.0.0.0&token=-32:1245372967:0:1343222881000:1343222711330
ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
Exception in doCheckpoint:
ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
java.io.FileNotFoundException:
http://AAA.xxx.yyy.zzz:50070/getimage?putimage=1&port=50090&machine=0.0.0.0&token=-32:1245372967:0:1343222881000:1343222711330
Where Primary NN is AAA.xxx.yyy.zzz.
The 0.0.0.0 embedded in the URL looks suspicious, but I have no idea
what file it is telling me it is missing.
On 08/15/2012 12:39 PM, Terry Healy wrote:
> This situation continues and reports every 10 minutes. I even tried
> moving the secondary NN function to a different node. I have run TCPDump
> but cannot isolate the "Connection Refused" issue. Am I correct in
> assuming that the NN will try to connect to SNN on port 50090?
>
> Any way out? Current partial dump below.
>
> 2012-08-15 12:36:09,422 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> xxx.yyy.254.254
>
> 2012-08-15 12:36:09,423 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit
> log, edits.new files already exists in all healthy directories:
> /home/thealy/hdfs/name/current/edits.new
>
> 2012-08-15 12:36:09,879 ERROR
> org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:thealy cause:java.net.ConnectException:
> Connection refused
>
> 2012-08-15 12:36:09,879 ERROR
> org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:thealy cause:java.net.ConnectException:
> Connection refused
>
> 2012-08-15 12:36:09,881 WARN org.mortbay.log: /getimage:
> java.io.IOException: GetImage failed. java.net.ConnectException:
> Connection refused
>
>
>
> On 07/09/2012 01:23 PM, Terry Healy wrote:
>> Any suggestions on how to clear the error? stp-all / start-all had no
>> effect.
>>
>> On 07/03/2012 04:22 PM, Brandon Li wrote:
>>> With 1.0.2, only one checkpoint process is executed at a time. When the
>>> namenode gets an overlapping checkpointing request, it checks edit.new
>>> in its storage directories. If all of them have this file, namenode
>>> concludes the previous checkpoint process is not done yet and prints the
>>> warning message you've seen.
>>>
>>> Brandon
>>>
>>> On Tue, Jul 3, 2012 at 10:56 AM, Terry Healy <thealy@bnl.gov
>>> <ma...@bnl.gov>> wrote:
>>>
>>> Running Apache 1.0.2.
>>>
>>> The NN log is reporting that it cannot "roll the edit log" from the
>>> secondary NN. The SecondaryNameNode is running on the system referred to
>>> as xxx.yyy.254.238 in the log snippet below.
>>>
>>> From the NN, I can connect to the Secondary via ssh as the user. Any
>>> suggestions what have I got wrong here?
>>>
>>> thanks,
>>>
>>> Terry
>>>
>>> INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>>> from xxx.yyy.254.238
>>> WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll
>>> edit log, edits.new files already exists in all healthy directories:
>>> /home/[user]/hdfs/name/current/edits.new
>>>
>>> ERROR org.apache.hadoop.security.UserGroupInformation:
>>> PriviledgedActionException as:[user] cause:java.net.ConnectException:
>>> Connection refused
>>> ERROR org.apache.hadoop.security.UserGroupInformation:
>>> PriviledgedActionException as:[user] cause:java.net.ConnectException:
>>> Connection refused
>>>
>>> WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed.
>>> java.net.ConnectException: Connection refused
>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
>>> ....
>>>
>>>
>>> --
>>> Terry Healy / thealy@bnl.gov <ma...@bnl.gov>
>>> Cyber Security Operations
>>> Brookhaven National Laboratory
>>> Building 515, Upton N.Y. 11973
>>>
>>>
>>
>
--
Terry Healy / thealy@bnl.gov
Cyber Security Operations
Brookhaven National Laboratory
Building 515, Upton N.Y. 11973