You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Mohit Vadhera <pr...@gmail.com> on 2013/03/29 06:54:56 UTC

Filesystem error

Hi,

I have filsystem error. when i run fsck to move corrupted blocks i get the
following error after stopping services i get the below error. but if i
don't start the services and run the fsck command the corrupted block
doesn't move.  I am not getting this Usergroupinformation error. it is
looking permission error. Can any body fix it . It is an urgent issue on my
hadoop machine. It is a standalone cluster configured using the below link

https://ccp.cloudera.com/display/CDH4DOC/Installing+CDH4+on+a+Single+Linux+Node+in+Pseudo-distributed+Mode

Error
===================================================

# sudo -u hdfs  hadoop fsck / -move
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

13/03/29 01:20:20 ERROR security.UserGroupInformation:
PriviledgedActionException as:hdfs (auth:SIMPLE)
cause:java.net.ConnectException: Call From OPERA-MAST1.ny.os.local/
172.20.3.119 to localhost:8020 failed on connection exception:
java.net.ConnectException: Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
Exception in thread "main" java.net.ConnectException: Call From
OPERA-MAST1.ny.os.local/172.20.3.119 to localhost:8020 failed on connection
exception: java.net.ConnectException: Connection refused; For more details
see:  http://wiki.apache.org/hadoop/ConnectionRefused
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:721)
        at org.apache.hadoop.ipc.Client.call(Client.java:1228)
        at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
        at $Proxy9.getFileInfo(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
        at $Proxy9.getFileInfo(Unknown Source)
        at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:628)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1507)
        at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:783)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1257)
        at org.apache.hadoop.hdfs.HAUtil.getAddressOfActive(HAUtil.java:298)
        at
org.apache.hadoop.hdfs.tools.DFSck.getCurrentNamenodeAddress(DFSck.java:229)
        at org.apache.hadoop.hdfs.tools.DFSck.doWork(DFSck.java:235)
        at org.apache.hadoop.hdfs.tools.DFSck.access$000(DFSck.java:71)
        at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:134)
        at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:131)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
        at org.apache.hadoop.hdfs.tools.DFSck.run(DFSck.java:130)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.hadoop.hdfs.tools.DFSck.main(DFSck.java:316)
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
        at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:523)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:488)
        at
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:498)
        at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:592)
        at
org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:240)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1277)
        at org.apache.hadoop.ipc.Client.call(Client.java:1195)
        ... 26 more

Re: Filesystem error

Posted by Steve Loughran <st...@apache.org>.
On 29 March 2013 05:54, Mohit Vadhera <pr...@gmail.com> wrote:

> Hi,
>
> I have filsystem error. when i run fsck to move corrupted blocks i get the
> following error after stopping services i get the below error. but if i
> don't start the services and run the fsck command the corrupted block
> doesn't move.  I am not getting this Usergroupinformation error. it is
> looking permission error. Can any body fix it . It is an urgent issue on my
> hadoop machine. It is a standalone cluster configured using the below link
>
>
> https://ccp.cloudera.com/display/CDH4DOC/Installing+CDH4+on+a+Single+Linux+Node+in+Pseudo-distributed+Mode
>


1. If you've got problems w/ any non-ASF Hadoop product -here the Cloudera
one- please take it up through their support channels and forums

http://wiki.apache.org/hadoop/InvalidJiraIssues
13/03/29 01:20:20 ERROR security.UserGroupInformation:

> PriviledgedActionException as:hdfs (auth:SIMPLE)
> cause:java.net.ConnectException: Call From OPERA-MAST1.ny.os.local/
> 172.20.3.119 to localhost:8020 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused
> Exception in thread "main" java.net.ConnectException: Call From
> OPERA-MAST1.ny.os.local/172.20.3.119 to localhost:8020 failed on
> connection
> exception: java.net.ConnectException: Connection refused; For more details
> see:  http://wiki.apache.org/hadoop/ConnectionRefused
>         at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:721)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1228)
>         at
>

That's an error message that I actually added to avoid support calls by
pointing to a  wiki page that actually explains what the message means and
how you need to go about diagnosing the problem on your installation.

Did you actually read the error message, see the referenced wiki page and
follow it?

1. If you didn't recognise that there was a link to a self-help page, I'd
love to get your recommendations as to how we could make it clearer to
people such as yourself that there is a URL in the message, that this page
should be the first place that you should go to to start diagnosing things
better. Should we drop the stack trace and just print in capital letters

STOP: GO TO THE WIKI PAGE http://wiki.apache.org/hadoop/ConnectionRefused

Because I'm not sure what else we could do?

We could try that, but the more detailed stack trace is designed for people
who do know more about hadoop internals, including those commercial support
teams, We can't drop the details without removing the escalation options
available to you.

2. If you did go to the wiki page, did you follow its step-by-step
instructions? If not: why not? Is there any way we could make these
instructions clearer and easier to follow? Do you think there's something
that is missing?

The ASF projects are, apart from those supported vendor channels, entire
self supporting through the community. We do try our utmost to help people
fend for themselves, which is why the error message includes the URL to a
diagnostics page, and the page has instructions. If this process is somehow
failing you, then I'd love some suggestions as to how it could be improved.

-Steve

Re: Filesystem error

Posted by Daryn Sharp <da...@yahoo-inc.com>.
If you are using CDH4, you will need to contact Cloudera support directly.  If you are just following their setup instructions, what version of hadoop are you running?  Have you scoured the NN and DN logs for unusual exceptions?  Short of losing every replica of a block, filesystem corruption should not occur.

Daryn

On Mar 29, 2013, at 10:53 AM, Mohit Vadhera wrote:

> Thanks, I started services and moved corrupted block files and it worked.
> but users are complaining they have lost data that i didn't move. this is
> very unfortunate i didn't move those file that they are complaining lost.
> 
> 2nd thing, This filesystem corruption happened second time. Can you please
> let me know what can be the reason of corruption of filesystem second time.
> 
> Can anbybody answer this.?
> 
> 
> On Fri, Mar 29, 2013 at 6:56 PM, Daryn Sharp <da...@yahoo-inc.com> wrote:
> 
>> The UGI preface is just reporting who you are when the exception occurred.
>> The issue isn't permissions but rather when you stopped the services it
>> can't connect to localhost:8020 because nothing is listening on 8020, hence
>> the "connection refused".  I think you need to force the NN into safe mode
>> rather than stop the services.
>> 
>> Daryn
>> 
>> On Mar 29, 2013, at 12:54 AM, Mohit Vadhera wrote:
>> 
>>> Hi,
>>> 
>>> I have filsystem error. when i run fsck to move corrupted blocks i get
>> the
>>> following error after stopping services i get the below error. but if i
>>> don't start the services and run the fsck command the corrupted block
>>> doesn't move.  I am not getting this Usergroupinformation error. it is
>>> looking permission error. Can any body fix it . It is an urgent issue on
>> my
>>> hadoop machine. It is a standalone cluster configured using the below
>> link
>>> 
>>> 
>> https://ccp.cloudera.com/display/CDH4DOC/Installing+CDH4+on+a+Single+Linux+Node+in+Pseudo-distributed+Mode
>>> 
>>> Error
>>> ===================================================
>>> 
>>> # sudo -u hdfs  hadoop fsck / -move
>>> DEPRECATED: Use of this script to execute hdfs command is deprecated.
>>> Instead use the hdfs command for it.
>>> 
>>> 13/03/29 01:20:20 ERROR security.UserGroupInformation:
>>> PriviledgedActionException as:hdfs (auth:SIMPLE)
>>> cause:java.net.ConnectException: Call From OPERA-MAST1.ny.os.local/
>>> 172.20.3.119 to localhost:8020 failed on connection exception:
>>> java.net.ConnectException: Connection refused; For more details see:
>>> http://wiki.apache.org/hadoop/ConnectionRefused
>>> Exception in thread "main" java.net.ConnectException: Call From
>>> OPERA-MAST1.ny.os.local/172.20.3.119 to localhost:8020 failed on
>> connection
>>> exception: java.net.ConnectException: Connection refused; For more
>> details
>>> see:  http://wiki.apache.org/hadoop/ConnectionRefused
>>>       at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:721)
>>>       at org.apache.hadoop.ipc.Client.call(Client.java:1228)
>>>       at
>>> 
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
>>>       at $Proxy9.getFileInfo(Unknown Source)
>>>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>       at
>>> 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>       at
>>> 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>       at java.lang.reflect.Method.invoke(Method.java:597)
>>>       at
>>> 
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
>>>       at
>>> 
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
>>>       at $Proxy9.getFileInfo(Unknown Source)
>>>       at
>>> 
>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:628)
>>>       at
>> org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1507)
>>>       at
>>> 
>> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:783)
>>>       at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1257)
>>>       at
>> org.apache.hadoop.hdfs.HAUtil.getAddressOfActive(HAUtil.java:298)
>>>       at
>>> 
>> org.apache.hadoop.hdfs.tools.DFSck.getCurrentNamenodeAddress(DFSck.java:229)
>>>       at org.apache.hadoop.hdfs.tools.DFSck.doWork(DFSck.java:235)
>>>       at org.apache.hadoop.hdfs.tools.DFSck.access$000(DFSck.java:71)
>>>       at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:134)
>>>       at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:131)
>>>       at java.security.AccessController.doPrivileged(Native Method)
>>>       at javax.security.auth.Subject.doAs(Subject.java:396)
>>>       at
>>> 
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
>>>       at org.apache.hadoop.hdfs.tools.DFSck.run(DFSck.java:130)
>>>       at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>       at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>       at org.apache.hadoop.hdfs.tools.DFSck.main(DFSck.java:316)
>>> Caused by: java.net.ConnectException: Connection refused
>>>       at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>       at
>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
>>>       at
>>> 
>> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207)
>>>       at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:523)
>>>       at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:488)
>>>       at
>>> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:498)
>>>       at
>>> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:592)
>>>       at
>>> org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:240)
>>>       at org.apache.hadoop.ipc.Client.getConnection(Client.java:1277)
>>>       at org.apache.hadoop.ipc.Client.call(Client.java:1195)
>>>       ... 26 more
>> 
>> 


Re: Filesystem error

Posted by Mohit Vadhera <pr...@gmail.com>.
Thanks, I started services and moved corrupted block files and it worked.
but users are complaining they have lost data that i didn't move. this is
very unfortunate i didn't move those file that they are complaining lost.

2nd thing, This filesystem corruption happened second time. Can you please
let me know what can be the reason of corruption of filesystem second time.

Can anbybody answer this.?


On Fri, Mar 29, 2013 at 6:56 PM, Daryn Sharp <da...@yahoo-inc.com> wrote:

> The UGI preface is just reporting who you are when the exception occurred.
>  The issue isn't permissions but rather when you stopped the services it
> can't connect to localhost:8020 because nothing is listening on 8020, hence
> the "connection refused".  I think you need to force the NN into safe mode
> rather than stop the services.
>
> Daryn
>
> On Mar 29, 2013, at 12:54 AM, Mohit Vadhera wrote:
>
> > Hi,
> >
> > I have filsystem error. when i run fsck to move corrupted blocks i get
> the
> > following error after stopping services i get the below error. but if i
> > don't start the services and run the fsck command the corrupted block
> > doesn't move.  I am not getting this Usergroupinformation error. it is
> > looking permission error. Can any body fix it . It is an urgent issue on
> my
> > hadoop machine. It is a standalone cluster configured using the below
> link
> >
> >
> https://ccp.cloudera.com/display/CDH4DOC/Installing+CDH4+on+a+Single+Linux+Node+in+Pseudo-distributed+Mode
> >
> > Error
> > ===================================================
> >
> > # sudo -u hdfs  hadoop fsck / -move
> > DEPRECATED: Use of this script to execute hdfs command is deprecated.
> > Instead use the hdfs command for it.
> >
> > 13/03/29 01:20:20 ERROR security.UserGroupInformation:
> > PriviledgedActionException as:hdfs (auth:SIMPLE)
> > cause:java.net.ConnectException: Call From OPERA-MAST1.ny.os.local/
> > 172.20.3.119 to localhost:8020 failed on connection exception:
> > java.net.ConnectException: Connection refused; For more details see:
> > http://wiki.apache.org/hadoop/ConnectionRefused
> > Exception in thread "main" java.net.ConnectException: Call From
> > OPERA-MAST1.ny.os.local/172.20.3.119 to localhost:8020 failed on
> connection
> > exception: java.net.ConnectException: Connection refused; For more
> details
> > see:  http://wiki.apache.org/hadoop/ConnectionRefused
> >        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:721)
> >        at org.apache.hadoop.ipc.Client.call(Client.java:1228)
> >        at
> >
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
> >        at $Proxy9.getFileInfo(Unknown Source)
> >        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >        at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >        at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >        at java.lang.reflect.Method.invoke(Method.java:597)
> >        at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
> >        at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
> >        at $Proxy9.getFileInfo(Unknown Source)
> >        at
> >
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:628)
> >        at
> org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1507)
> >        at
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:783)
> >        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1257)
> >        at
> org.apache.hadoop.hdfs.HAUtil.getAddressOfActive(HAUtil.java:298)
> >        at
> >
> org.apache.hadoop.hdfs.tools.DFSck.getCurrentNamenodeAddress(DFSck.java:229)
> >        at org.apache.hadoop.hdfs.tools.DFSck.doWork(DFSck.java:235)
> >        at org.apache.hadoop.hdfs.tools.DFSck.access$000(DFSck.java:71)
> >        at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:134)
> >        at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:131)
> >        at java.security.AccessController.doPrivileged(Native Method)
> >        at javax.security.auth.Subject.doAs(Subject.java:396)
> >        at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
> >        at org.apache.hadoop.hdfs.tools.DFSck.run(DFSck.java:130)
> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> >        at org.apache.hadoop.hdfs.tools.DFSck.main(DFSck.java:316)
> > Caused by: java.net.ConnectException: Connection refused
> >        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >        at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
> >        at
> >
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207)
> >        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:523)
> >        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:488)
> >        at
> > org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:498)
> >        at
> > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:592)
> >        at
> > org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:240)
> >        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1277)
> >        at org.apache.hadoop.ipc.Client.call(Client.java:1195)
> >        ... 26 more
>
>

Re: Filesystem error

Posted by Daryn Sharp <da...@yahoo-inc.com>.
The UGI preface is just reporting who you are when the exception occurred.  The issue isn't permissions but rather when you stopped the services it can't connect to localhost:8020 because nothing is listening on 8020, hence the "connection refused".  I think you need to force the NN into safe mode rather than stop the services.

Daryn

On Mar 29, 2013, at 12:54 AM, Mohit Vadhera wrote:

> Hi,
> 
> I have filsystem error. when i run fsck to move corrupted blocks i get the
> following error after stopping services i get the below error. but if i
> don't start the services and run the fsck command the corrupted block
> doesn't move.  I am not getting this Usergroupinformation error. it is
> looking permission error. Can any body fix it . It is an urgent issue on my
> hadoop machine. It is a standalone cluster configured using the below link
> 
> https://ccp.cloudera.com/display/CDH4DOC/Installing+CDH4+on+a+Single+Linux+Node+in+Pseudo-distributed+Mode
> 
> Error
> ===================================================
> 
> # sudo -u hdfs  hadoop fsck / -move
> DEPRECATED: Use of this script to execute hdfs command is deprecated.
> Instead use the hdfs command for it.
> 
> 13/03/29 01:20:20 ERROR security.UserGroupInformation:
> PriviledgedActionException as:hdfs (auth:SIMPLE)
> cause:java.net.ConnectException: Call From OPERA-MAST1.ny.os.local/
> 172.20.3.119 to localhost:8020 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused
> Exception in thread "main" java.net.ConnectException: Call From
> OPERA-MAST1.ny.os.local/172.20.3.119 to localhost:8020 failed on connection
> exception: java.net.ConnectException: Connection refused; For more details
> see:  http://wiki.apache.org/hadoop/ConnectionRefused
>        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:721)
>        at org.apache.hadoop.ipc.Client.call(Client.java:1228)
>        at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
>        at $Proxy9.getFileInfo(Unknown Source)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
>        at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
>        at $Proxy9.getFileInfo(Unknown Source)
>        at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:628)
>        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1507)
>        at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:783)
>        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1257)
>        at org.apache.hadoop.hdfs.HAUtil.getAddressOfActive(HAUtil.java:298)
>        at
> org.apache.hadoop.hdfs.tools.DFSck.getCurrentNamenodeAddress(DFSck.java:229)
>        at org.apache.hadoop.hdfs.tools.DFSck.doWork(DFSck.java:235)
>        at org.apache.hadoop.hdfs.tools.DFSck.access$000(DFSck.java:71)
>        at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:134)
>        at org.apache.hadoop.hdfs.tools.DFSck$1.run(DFSck.java:131)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at javax.security.auth.Subject.doAs(Subject.java:396)
>        at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
>        at org.apache.hadoop.hdfs.tools.DFSck.run(DFSck.java:130)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>        at org.apache.hadoop.hdfs.tools.DFSck.main(DFSck.java:316)
> Caused by: java.net.ConnectException: Connection refused
>        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>        at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
>        at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207)
>        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:523)
>        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:488)
>        at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:498)
>        at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:592)
>        at
> org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:240)
>        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1277)
>        at org.apache.hadoop.ipc.Client.call(Client.java:1195)
>        ... 26 more