You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "Jitendra Nath Pandey (JIRA)" <ji...@apache.org> on 2019/03/05 22:09:01 UTC
[jira] [Resolved] (HDDS-725) Exception thrown in loop while trying
to write a file in ozonefs
[ https://issues.apache.org/jira/browse/HDDS-725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jitendra Nath Pandey resolved HDDS-725.
---------------------------------------
Resolution: Fixed
Fix Version/s: 0.4.0
I think this has been fixed. Please re-open if the issue re-surfaces.
> Exception thrown in loop while trying to write a file in ozonefs
> ----------------------------------------------------------------
>
> Key: HDDS-725
> URL: https://issues.apache.org/jira/browse/HDDS-725
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Manager
> Affects Versions: 0.3.0
> Environment:
>
> Reporter: Nilotpal Nandi
> Priority: Blocker
> Labels: test-badlands
> Fix For: 0.4.0
>
> Attachments: all-node-ozone-logs-1540375264.tar.gz
>
>
> Ran the following command :
> ----------------------------------------
> ozone fs -put 2GB /testdir5/
> Exceptions are thrown continuously in loop. Please note that there are 8 datanodes alive in the cluster.
> {noformat}
> root@ctr-e138-1518143905142-544443-01-000008 logs]# /root/allssh.sh 'jps -l | grep Datanode'
> ------------------------
> Host::172.27.20.96
> ------------------------
> 411564 org.apache.hadoop.ozone.HddsDatanodeService
> ------------------------
> Host::172.27.20.91
> ------------------------
> 472897 org.apache.hadoop.ozone.HddsDatanodeService
> ------------------------
> Host::172.27.38.9
> ------------------------
> 351139 org.apache.hadoop.ozone.HddsDatanodeService
> ------------------------
> Host::172.27.24.90
> ------------------------
> 314304 org.apache.hadoop.ozone.HddsDatanodeService
> ------------------------
> Host::172.27.15.139
> ------------------------
> 324820 org.apache.hadoop.ozone.HddsDatanodeService
> ------------------------
> Host::172.27.10.199
> ------------------------
> ------------------------
> Host::172.27.15.131
> ------------------------
> ------------------------
> Host::172.27.57.0
> ------------------------
> ------------------------
> Host::172.27.23.139
> ------------------------
> 627053 org.apache.hadoop.ozone.HddsDatanodeService
> ------------------------
> Host::172.27.68.65
> ------------------------
> 557443 org.apache.hadoop.ozone.HddsDatanodeService
> ------------------------
> Host::172.27.19.74
> ------------------------
> ------------------------
> Host::172.27.85.64
> ------------------------
> 508121 org.apache.hadoop.ozone.HddsDatanodeService{noformat}
>
> {noformat}
>
> 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.LeaderElection: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: Election REJECTED; received 0 response(s) [] and 2 exception(s); 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57:t16296, leader=null, voted=7c3b2fb1-cf16-4e5f-94dc-8a089492ad57, raftlog=[(t:37, i:271)], conf=271: [7c3b2fb1-cf16-4e5f-94dc-8a089492ad57:172.27.85.64:9858, 86f9e313-ae49-4675-95d7-27856641aee1:172.27.15.131:9858, 9524f4e2-9031-4852-ab7c-11c2da3460db:172.27.57.0:9858], old=null
> 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.LeaderElection: 0: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.LeaderElection: 1: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.RaftServerImpl: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57 changes role from CANDIDATE to FOLLOWER at term 16296 for changeToFollower
> 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.RoleInfo: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: shutdown LeaderElection
> 2018-10-24 09:49:47,093 INFO org.apache.ratis.server.impl.RoleInfo: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: start FollowerState
> 2018-10-24 09:49:48,171 INFO org.apache.ratis.server.impl.FollowerState: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57 changes to CANDIDATE, lastRpcTime:1078, electionTimeout:1078ms
> 2018-10-24 09:49:48,171 INFO org.apache.ratis.server.impl.RoleInfo: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: shutdown FollowerState
> 2018-10-24 09:49:48,171 INFO org.apache.ratis.server.impl.RaftServerImpl: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57 changes role from FOLLOWER to CANDIDATE at term 16296 for changeToCandidate
> 2018-10-24 09:49:48,172 INFO org.apache.ratis.server.impl.RoleInfo: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: start LeaderElection
> 2018-10-24 09:49:48,173 INFO org.apache.ratis.server.impl.LeaderElection: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57: begin an election in Term 16297
> 2018-10-24 09:49:48,174 INFO org.apache.ratis.server.impl.LeaderElection: 7c3b2fb1-cf16-4e5f-94dc-8a089492ad57 got exception when requesting votes: {}
> java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:214)
> at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:146)
> at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:102)
> Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:222)
> at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:203)
> at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:132)
> at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265)
> at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:61)
> at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:150)
> at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$0(LeaderElection.java:188)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.27.15.131:9858
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:325)
> at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
> at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:634)
> at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)
> at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
> at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
> at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
> at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> ... 1 more
> Caused by: java.net.ConnectException: Connection refused
> ... 11 more
> {noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org