You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "liaowenrui (JIRA)" <ji...@apache.org> on 2013/01/15 03:00:14 UTC

[jira] [Commented] (HDFS-4404) create file failure,when the machine of firse NameNod is down

    [ https://issues.apache.org/jira/browse/HDFS-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13553401#comment-13553401 ] 

liaowenrui commented on HDFS-4404:
----------------------------------

failure reason is that create file marked as not idempotent.

org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create. Not retrying because the invoked method is not idempotent, and unable to determine whether it was invoked

here is not related to BK.

log:

013-01-12 09:52:47,834 WARN retry.RetryInvocationHandler (RetryInvocationHandler.java:invoke(95)) - Exception while invoking class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create. Not retrying because the invoked method is not idempotent, and unable to determine whether it was invoked

when client create file,the client will connect the first NameNode,but client don't connect to the first NameNode,because machine of first NameNode is shutdown.the create function mark as not idempotent,and create file op don't retry.so the create file op is failure. I think that when client don't connect to namenode,not idempotent will be retry. 

                
> create file failure,when the machine of firse NameNod is down
> -------------------------------------------------------------
>
>                 Key: HDFS-4404
>                 URL: https://issues.apache.org/jira/browse/HDFS-4404
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs-client
>    Affects Versions: 2.0.2-alpha
>            Reporter: liaowenrui
>            Priority: Blocker
>             Fix For: 2.0.2-alpha, 2.0.3-alpha
>
>
> test Environment: NN1,NN2,DN1,DN2,DN3
> machine1:NN1,DN1
> machine2:NN2,DN2
> machine3:DN3
> mathine1 is down.
> 2013-01-12 09:51:21,248 DEBUG ipc.Client (Client.java:setupIOstreams(562)) - Connecting to /160.161.0.155:8020
> 2013-01-12 09:51:38,442 DEBUG ipc.Client (Client.java:close(932)) - closing ipc connection to vm2/160.161.0.155:8020: 10000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/160.161.0.155:8020]
> java.net.SocketTimeoutException: 10000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/160.161.0.155:8020]
>  at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213)
>  at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:524)
>  at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489)
>  at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:474)
>  at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:568)
>  at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:217)
>  at org.apache.hadoop.ipc.Client.getConnection(Client.java:1286)
>  at org.apache.hadoop.ipc.Client.call(Client.java:1156)
>  at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184)
>  at $Proxy9.create(Unknown Source)
>  at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:187)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
>  at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:165)
>  at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:84)
>  at $Proxy10.create(Unknown Source)
>  at org.apache.hadoop.hdfs.DFSOutputStream.<init>(DFSOutputStream.java:1261)
>  at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1280)
>  at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1128)
>  at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1086)
>  at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:232)
>  at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:75)
>  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:806)
>  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787)
>  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:715)
>  at test.TestLease.main(TestLease.java:45)
> 2013-01-12 09:51:38,443 DEBUG ipc.Client (Client.java:close(940)) - IPC Client (31594013) connection to /160.161.0.155:8020 from hdfs/hadoop@HADOOP.COM: closed
> 2013-01-12 09:52:47,834 WARN  retry.RetryInvocationHandler (RetryInvocationHandler.java:invoke(95)) - Exception while invoking class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create. Not retrying because the invoked method is not idempotent, and unable to determine whether it was invoked
> java.net.SocketTimeoutException: Call From szxy1x001833091/172.0.0.13 to vm2:8020 failed on socket timeout exception: java.net.SocketTimeoutException: 10000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/160.161.0.155:8020]; For more details see:  http://wiki.apache.org/hadoop/SocketTimeout
>  at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:743)
>  at org.apache.hadoop.ipc.Client.call(Client.java:1180)
>  at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184)
>  at $Proxy9.create(Unknown Source)
>  at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:187)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
>  at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:165)
>  at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:84)
>  at $Proxy10.create(Unknown Source)
>  at org.apache.hadoop.hdfs.DFSOutputStream.<init>(DFSOutputStream.java:1261)
>  at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1280)
>  at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1128)
>  at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1086)
>  at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:232)
>  at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:75)
>  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:806)
>  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787)
>  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:715)
>  at test.TestLease.main(TestLease.java:45)
> Caused by: java.net.SocketTimeoutException: 10000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/160.161.0.155:8020]
>  at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213)
>  at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:524)
>  at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489)
>  at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:474)
>  at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:568)
>  at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:217)
>  at org.apache.hadoop.ipc.Client.getConnection(Client.java:1286)
>  at org.apache.hadoop.ipc.Client.call(Client.java:1156)
>  ... 20 more
> java.net.SocketTimeoutException: Call From szxy1x001833091/172.0.0.13 to vm2:8020 failed on socket timeout exception: java.net.SocketTimeoutException: 10000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/160.161.0.155:8020]; For more details see:  http://wiki.apache.org/hadoop/SocketTimeout
>  at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:743)
>  at org.apache.hadoop.ipc.Client.call(Client.java:1180)
>  at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184)
>  at $Proxy9.create(Unknown Source)
>  at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:187)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
>  at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:165)
>  at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:84)
>  at $Proxy10.create(Unknown Source)
>  at org.apache.hadoop.hdfs.DFSOutputStream.<init>(DFSOutputStream.java:1261)
>  at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1280)
>  at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1128)
>  at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1086)
>  at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:232)
>  at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:75)
>  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:806)
>  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787)
>  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:715)
>  at test.TestLease.main(TestLease.java:45)
> Caused by: java.net.SocketTimeoutException: 10000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/160.161.0.155:8020]
>  at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:213)
>  at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:524)
>  at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489)
>  at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:474)
>  at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:568)
>  at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:217)
>  at org.apache.hadoop.ipc.Client.getConnection(Client.java:1286)
>  at org.apache.hadoop.ipc.Client.call(Client.java:1156)
>  ... 20 more
> 2013-01-12 09:54:52,269 DEBUG ipc.Client (Client.java:stop(1021)) - Stopping client

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira