You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Terry Healy <th...@bnl.gov> on 2012/07/03 19:56:04 UTC
'Cannot roll edit log'
Running Apache 1.0.2.
The NN log is reporting that it cannot "roll the edit log" from the
secondary NN. The SecondaryNameNode is running on the system referred to
as xxx.yyy.254.238 in the log snippet below.
>From the NN, I can connect to the Secondary via ssh as the user. Any
suggestions what have I got wrong here?
thanks,
Terry
INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
from xxx.yyy.254.238
WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll
edit log, edits.new files already exists in all healthy directories:
/home/[user]/hdfs/name/current/edits.new
ERROR org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:[user] cause:java.net.ConnectException:
Connection refused
ERROR org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:[user] cause:java.net.ConnectException:
Connection refused
WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed.
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
....
--
Terry Healy / thealy@bnl.gov
Cyber Security Operations
Brookhaven National Laboratory
Building 515, Upton N.Y. 11973
Re: 'Cannot roll edit log'
Posted by Terry Healy <th...@bnl.gov>.
Further info: From the log of secondary NN:
INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Posted
URL
AAA.xxx.yyy.zzz:50070putimage=1&port=50090&machine=0.0.0.0&token=-32:1245372967:0:1343222881000:1343222711330
ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
Exception in doCheckpoint:
ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
java.io.FileNotFoundException:
http://AAA.xxx.yyy.zzz:50070/getimage?putimage=1&port=50090&machine=0.0.0.0&token=-32:1245372967:0:1343222881000:1343222711330
Where Primary NN is AAA.xxx.yyy.zzz.
The 0.0.0.0 embedded in the URL looks suspicious, but I have no idea
what file it is telling me it is missing.
On 08/15/2012 12:39 PM, Terry Healy wrote:
> This situation continues and reports every 10 minutes. I even tried
> moving the secondary NN function to a different node. I have run TCPDump
> but cannot isolate the "Connection Refused" issue. Am I correct in
> assuming that the NN will try to connect to SNN on port 50090?
>
> Any way out? Current partial dump below.
>
> 2012-08-15 12:36:09,422 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
> xxx.yyy.254.254
>
> 2012-08-15 12:36:09,423 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit
> log, edits.new files already exists in all healthy directories:
> /home/thealy/hdfs/name/current/edits.new
>
> 2012-08-15 12:36:09,879 ERROR
> org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:thealy cause:java.net.ConnectException:
> Connection refused
>
> 2012-08-15 12:36:09,879 ERROR
> org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:thealy cause:java.net.ConnectException:
> Connection refused
>
> 2012-08-15 12:36:09,881 WARN org.mortbay.log: /getimage:
> java.io.IOException: GetImage failed. java.net.ConnectException:
> Connection refused
>
>
>
> On 07/09/2012 01:23 PM, Terry Healy wrote:
>> Any suggestions on how to clear the error? stp-all / start-all had no
>> effect.
>>
>> On 07/03/2012 04:22 PM, Brandon Li wrote:
>>> With 1.0.2, only one checkpoint process is executed at a time. When the
>>> namenode gets an overlapping checkpointing request, it checks edit.new
>>> in its storage directories. If all of them have this file, namenode
>>> concludes the previous checkpoint process is not done yet and prints the
>>> warning message you've seen.
>>>
>>> Brandon
>>>
>>> On Tue, Jul 3, 2012 at 10:56 AM, Terry Healy <thealy@bnl.gov
>>> <ma...@bnl.gov>> wrote:
>>>
>>> Running Apache 1.0.2.
>>>
>>> The NN log is reporting that it cannot "roll the edit log" from the
>>> secondary NN. The SecondaryNameNode is running on the system referred to
>>> as xxx.yyy.254.238 in the log snippet below.
>>>
>>> From the NN, I can connect to the Secondary via ssh as the user. Any
>>> suggestions what have I got wrong here?
>>>
>>> thanks,
>>>
>>> Terry
>>>
>>> INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>>> from xxx.yyy.254.238
>>> WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll
>>> edit log, edits.new files already exists in all healthy directories:
>>> /home/[user]/hdfs/name/current/edits.new
>>>
>>> ERROR org.apache.hadoop.security.UserGroupInformation:
>>> PriviledgedActionException as:[user] cause:java.net.ConnectException:
>>> Connection refused
>>> ERROR org.apache.hadoop.security.UserGroupInformation:
>>> PriviledgedActionException as:[user] cause:java.net.ConnectException:
>>> Connection refused
>>>
>>> WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed.
>>> java.net.ConnectException: Connection refused
>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
>>> ....
>>>
>>>
>>> --
>>> Terry Healy / thealy@bnl.gov <ma...@bnl.gov>
>>> Cyber Security Operations
>>> Brookhaven National Laboratory
>>> Building 515, Upton N.Y. 11973
>>>
>>>
>>
>
--
Terry Healy / thealy@bnl.gov
Cyber Security Operations
Brookhaven National Laboratory
Building 515, Upton N.Y. 11973
Re: 'Cannot roll edit log'
Posted by Terry Healy <th...@bnl.gov>.
This situation continues and reports every 10 minutes. I even tried
moving the secondary NN function to a different node. I have run TCPDump
but cannot isolate the "Connection Refused" issue. Am I correct in
assuming that the NN will try to connect to SNN on port 50090?
Any way out? Current partial dump below.
2012-08-15 12:36:09,422 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
xxx.yyy.254.254
2012-08-15 12:36:09,423 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll edit
log, edits.new files already exists in all healthy directories:
/home/thealy/hdfs/name/current/edits.new
2012-08-15 12:36:09,879 ERROR
org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:thealy cause:java.net.ConnectException:
Connection refused
2012-08-15 12:36:09,879 ERROR
org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:thealy cause:java.net.ConnectException:
Connection refused
2012-08-15 12:36:09,881 WARN org.mortbay.log: /getimage:
java.io.IOException: GetImage failed. java.net.ConnectException:
Connection refused
On 07/09/2012 01:23 PM, Terry Healy wrote:
> Any suggestions on how to clear the error? stp-all / start-all had no
> effect.
>
> On 07/03/2012 04:22 PM, Brandon Li wrote:
>> With 1.0.2, only one checkpoint process is executed at a time. When the
>> namenode gets an overlapping checkpointing request, it checks edit.new
>> in its storage directories. If all of them have this file, namenode
>> concludes the previous checkpoint process is not done yet and prints the
>> warning message you've seen.
>>
>> Brandon
>>
>> On Tue, Jul 3, 2012 at 10:56 AM, Terry Healy <thealy@bnl.gov
>> <ma...@bnl.gov>> wrote:
>>
>> Running Apache 1.0.2.
>>
>> The NN log is reporting that it cannot "roll the edit log" from the
>> secondary NN. The SecondaryNameNode is running on the system referred to
>> as xxx.yyy.254.238 in the log snippet below.
>>
>> From the NN, I can connect to the Secondary via ssh as the user. Any
>> suggestions what have I got wrong here?
>>
>> thanks,
>>
>> Terry
>>
>> INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>> from xxx.yyy.254.238
>> WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll
>> edit log, edits.new files already exists in all healthy directories:
>> /home/[user]/hdfs/name/current/edits.new
>>
>> ERROR org.apache.hadoop.security.UserGroupInformation:
>> PriviledgedActionException as:[user] cause:java.net.ConnectException:
>> Connection refused
>> ERROR org.apache.hadoop.security.UserGroupInformation:
>> PriviledgedActionException as:[user] cause:java.net.ConnectException:
>> Connection refused
>>
>> WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed.
>> java.net.ConnectException: Connection refused
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
>> ....
>>
>>
>> --
>> Terry Healy / thealy@bnl.gov <ma...@bnl.gov>
>> Cyber Security Operations
>> Brookhaven National Laboratory
>> Building 515, Upton N.Y. 11973
>>
>>
>
--
Terry Healy / thealy@bnl.gov
Cyber Security Operations
Brookhaven National Laboratory
Building 515, Upton N.Y. 11973
Re: 'Cannot roll edit log'
Posted by Terry Healy <th...@bnl.gov>.
Any suggestions on how to clear the error? stp-all / start-all had no
effect.
On 07/03/2012 04:22 PM, Brandon Li wrote:
> With 1.0.2, only one checkpoint process is executed at a time. When the
> namenode gets an overlapping checkpointing request, it checks edit.new
> in its storage directories. If all of them have this file, namenode
> concludes the previous checkpoint process is not done yet and prints the
> warning message you've seen.
>
> Brandon
>
> On Tue, Jul 3, 2012 at 10:56 AM, Terry Healy <thealy@bnl.gov
> <ma...@bnl.gov>> wrote:
>
> Running Apache 1.0.2.
>
> The NN log is reporting that it cannot "roll the edit log" from the
> secondary NN. The SecondaryNameNode is running on the system referred to
> as xxx.yyy.254.238 in the log snippet below.
>
> From the NN, I can connect to the Secondary via ssh as the user. Any
> suggestions what have I got wrong here?
>
> thanks,
>
> Terry
>
> INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from xxx.yyy.254.238
> WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll
> edit log, edits.new files already exists in all healthy directories:
> /home/[user]/hdfs/name/current/edits.new
>
> ERROR org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:[user] cause:java.net.ConnectException:
> Connection refused
> ERROR org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:[user] cause:java.net.ConnectException:
> Connection refused
>
> WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed.
> java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> ....
>
>
> --
> Terry Healy / thealy@bnl.gov <ma...@bnl.gov>
> Cyber Security Operations
> Brookhaven National Laboratory
> Building 515, Upton N.Y. 11973
>
>
--
Terry Healy / thealy@bnl.gov
Cyber Security Operations
Brookhaven National Laboratory
Building 515, Upton N.Y. 11973
Re: 'Cannot roll edit log'
Posted by Brandon Li <br...@hortonworks.com>.
With 1.0.2, only one checkpoint process is executed at a time. When the
namenode gets an overlapping checkpointing request, it checks edit.new in
its storage directories. If all of them have this file, namenode concludes
the previous checkpoint process is not done yet and prints the warning
message you've seen.
Brandon
On Tue, Jul 3, 2012 at 10:56 AM, Terry Healy <th...@bnl.gov> wrote:
> Running Apache 1.0.2.
>
> The NN log is reporting that it cannot "roll the edit log" from the
> secondary NN. The SecondaryNameNode is running on the system referred to
> as xxx.yyy.254.238 in the log snippet below.
>
> From the NN, I can connect to the Secondary via ssh as the user. Any
> suggestions what have I got wrong here?
>
> thanks,
>
> Terry
>
> INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> from xxx.yyy.254.238
> WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Cannot roll
> edit log, edits.new files already exists in all healthy directories:
> /home/[user]/hdfs/name/current/edits.new
>
> ERROR org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:[user] cause:java.net.ConnectException:
> Connection refused
> ERROR org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:[user] cause:java.net.ConnectException:
> Connection refused
>
> WARN org.mortbay.log: /getimage: java.io.IOException: GetImage failed.
> java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> ....
>
>
> --
> Terry Healy / thealy@bnl.gov
> Cyber Security Operations
> Brookhaven National Laboratory
> Building 515, Upton N.Y. 11973
>
>