You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Namit Maheshwari (JIRA)" <ji...@apache.org> on 2018/10/18 03:32:00 UTC
[jira] [Commented] (HDDS-689) Datanode shuts down on restart
[ https://issues.apache.org/jira/browse/HDDS-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16654583#comment-16654583 ]
Namit Maheshwari commented on HDDS-689:
---------------------------------------
{code:java}
2018-10-18 01:09:39,761 INFO datanode.DataNode (DataNode.java:startDataNode(1399)) - Starting DataNode with maxLockedMemory = 0
2018-10-18 01:09:39,790 INFO datanode.DataNode (DataNode.java:initDataXceiver(1147)) - Opened streaming server at /0.0.0.0:50010
2018-10-18 01:09:39,793 INFO datanode.DataNode (DataXceiverServer.java:<init>(78)) - Balancing bandwidth is 6250000 bytes/s
2018-10-18 01:09:39,793 INFO datanode.DataNode (DataXceiverServer.java:<init>(79)) - Number threads for balancing is 50
2018-10-18 01:09:39,796 INFO datanode.DataNode (DataXceiverServer.java:<init>(78)) - Balancing bandwidth is 6250000 bytes/s
2018-10-18 01:09:39,796 INFO datanode.DataNode (DataXceiverServer.java:<init>(79)) - Number threads for balancing is 50
2018-10-18 01:09:39,797 INFO datanode.DataNode (DataNode.java:initDataXceiver(1165)) - Listening on UNIX domain socket: /var/lib/hadoop-hdfs/dn_socket
2018-10-18 01:09:39,847 INFO util.log (Log.java:initialized(192)) - Logging initialized @2926ms
2018-10-18 01:09:39,975 INFO server.AuthenticationFilter (AuthenticationFilter.java:constructSecretProvider(240)) - Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
2018-10-18 01:09:39,979 INFO http.HttpRequestLog (HttpRequestLog.java:getRequestLog(81)) - Http request log for http.requests.datanode is not defined
2018-10-18 01:09:39,986 INFO http.HttpServer2 (HttpServer2.java:addGlobalFilter(968)) - Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2018-10-18 01:09:39,989 INFO http.HttpServer2 (HttpServer2.java:addFilter(941)) - Added filter authentication (class=org.apache.hadoop.security.authentication.server.AuthenticationFilter) to context datanode
2018-10-18 01:09:39,989 INFO http.HttpServer2 (HttpServer2.java:addFilter(951)) - Added filter authentication (class=org.apache.hadoop.security.authentication.server.AuthenticationFilter) to context logs
2018-10-18 01:09:39,990 INFO http.HttpServer2 (HttpServer2.java:addFilter(951)) - Added filter authentication (class=org.apache.hadoop.security.authentication.server.AuthenticationFilter) to context static
2018-10-18 01:09:39,990 INFO security.HttpCrossOriginFilterInitializer (HttpCrossOriginFilterInitializer.java:initFilter(49)) - CORS filter not enabled. Please set hadoop.http.cross-origin.enabled to 'true' to enable it
2018-10-18 01:09:40,018 INFO http.HttpServer2 (HttpServer2.java:bindListener(1185)) - Jetty bound to port 33229
2018-10-18 01:09:40,020 INFO server.Server (Server.java:doStart(346)) - jetty-9.3.19.v20170502
2018-10-18 01:09:40,059 INFO server.AuthenticationFilter (AuthenticationFilter.java:constructSecretProvider(240)) - Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
2018-10-18 01:09:40,062 INFO handler.ContextHandler (ContextHandler.java:doStart(781)) - Started o.e.j.s.ServletContextHandler@54a67a45{/logs,file:///grid/0/log/hdfs/hdfs/,AVAILABLE}
2018-10-18 01:09:40,062 INFO handler.ContextHandler (ContextHandler.java:doStart(781)) - Started o.e.j.s.ServletContextHandler@56aaaecd{/static,file:///usr/hdp/3.0.3.0-63/hadoop-hdfs/webapps/static/,AVAILABLE}
2018-10-18 01:09:40,147 INFO handler.ContextHandler (ContextHandler.java:doStart(781)) - Started o.e.j.w.WebAppContext@792bbc74{/,file:///usr/hdp/3.0.3.0-63/hadoop-hdfs/webapps/datanode/,AVAILABLE}{/datanode}
2018-10-18 01:09:40,155 INFO server.AbstractConnector (AbstractConnector.java:doStart(278)) - Started ServerConnector@2bb7bd00{HTTP/1.1,[http/1.1]}{localhost:33229}
2018-10-18 01:09:40,156 INFO server.Server (Server.java:doStart(414)) - Started @3235ms
2018-10-18 01:09:40,384 INFO web.DatanodeHttpServer (DatanodeHttpServer.java:start(255)) - Listening HTTP traffic on /0.0.0.0:50075
2018-10-18 01:09:40,391 INFO util.JvmPauseMonitor (JvmPauseMonitor.java:run(188)) - Starting JVM pause monitor
2018-10-18 01:09:40,396 INFO datanode.DataNode (DataNode.java:startDataNode(1427)) - dnUserName = hdfs
2018-10-18 01:09:40,396 INFO datanode.DataNode (DataNode.java:startDataNode(1428)) - supergroup = hdfs
2018-10-18 01:09:40,461 INFO ipc.CallQueueManager (CallQueueManager.java:<init>(84)) - Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 1000 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler
2018-10-18 01:09:40,531 INFO ipc.Server (Server.java:run(1074)) - Starting Socket Reader #1 for port 8010
2018-10-18 01:09:40,799 INFO datanode.DataNode (DataNode.java:initIpcServer(1033)) - Opened IPC server at /0.0.0.0:8010
2018-10-18 01:09:40,823 INFO datanode.DataNode (BlockPoolManager.java:refreshNamenodes(149)) - Refresh request received for nameservices: mycluster
2018-10-18 01:09:40,837 INFO datanode.DataNode (BlockPoolManager.java:doRefreshNamenodes(210)) - Starting BPOfferServices for nameservices: mycluster
2018-10-18 01:09:40,851 INFO datanode.DataNode (BPServiceActor.java:run(810)) - Block pool <registering> (Datanode Uuid unassigned) service to ctr-e138-1518143905142-510793-01-000004.hwx.site/172.27.79.197:8020 starting to offer service
2018-10-18 01:09:40,851 INFO datanode.DataNode (BPServiceActor.java:run(810)) - Block pool <registering> (Datanode Uuid unassigned) service to ctr-e138-1518143905142-510793-01-000005.hwx.site/172.27.87.64:8020 starting to offer service
2018-10-18 01:09:40,859 INFO ipc.Server (Server.java:run(1314)) - IPC Server Responder: starting
2018-10-18 01:09:40,859 INFO ipc.Server (Server.java:run(1153)) - IPC Server listener on 8010: starting
2018-10-18 01:09:40,894 INFO conf.Configuration (Configuration.java:removeUndeclaredTags(3298)) - Removed undeclared tags:
2018-10-18 01:09:40,958 INFO volume.VolumeUsage (VolumeUsage.java:loadScmUsed(136)) - Cached ScmUsed found for /grid/0/hadoop/hdfs/data : 3899052032
2018-10-18 01:09:40,961 INFO volume.HddsVolume (HddsVolume.java:<init>(147)) - Creating Volume: /grid/0/hadoop/hdfs/data/hdds of storage type : DISK and capacity : 270429917184
2018-10-18 01:09:40,967 INFO volume.VolumeSet (VolumeSet.java:initializeVolumeSet(140)) - Added Volume : /grid/0/hadoop/hdfs/data/hdds to VolumeSet
2018-10-18 01:09:40,992 INFO conf.Configuration (Configuration.java:removeUndeclaredTags(3298)) - Removed undeclared tags:
2018-10-18 01:09:41,122 INFO datanode.DataNode (BPOfferService.java:verifyAndSetNamespaceInfo(378)) - Acknowledging ACTIVE Namenode during handshakeBlock pool <registering> (Datanode Uuid unassigned) service to ctr-e138-1518143905142-510793-01-000005.hwx.site/172.27.87.64:8020
2018-10-18 01:09:41,127 INFO common.Storage (DataStorage.java:getParallelVolumeLoadThreadsNum(354)) - Using 1 threads to upgrade data directories (dfs.datanode.parallel.volumes.load.threads.num=1, dataDirs=1)
2018-10-18 01:09:41,137 INFO common.Storage (Storage.java:tryLock(905)) - Lock on /grid/0/hadoop/hdfs/data/in_use.lock acquired by nodename 3596688@ctr-e138-1518143905142-510793-01-000007.hwx.site
2018-10-18 01:09:41,174 INFO conf.Configuration (Configuration.java:removeUndeclaredTags(3298)) - Removed undeclared tags:
2018-10-18 01:09:41,176 INFO common.Storage (BlockPoolSliceStorage.java:recoverTransitionRead(251)) - Analyzing storage directories for bpid BP-1776487077-172.27.79.197-1539042949074
2018-10-18 01:09:41,176 INFO common.Storage (Storage.java:lock(864)) - Locking is disabled for /grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074
2018-10-18 01:09:41,179 INFO datanode.DataNode (DataNode.java:initStorage(1721)) - Setting up storage: nsid=99643741;bpid=BP-1776487077-172.27.79.197-1539042949074;lv=-57;nsInfo=lv=-64;cid=CID-2437e1cf-89ae-460e-95e7-996a9d7bccc2;nsid=99643741;c=1539042949074;bpid=BP-1776487077-172.27.79.197-1539042949074;dnuuid=3092b844-60f3-43f7-aaa7-18286872be3e
2018-10-18 01:09:41,323 INFO impl.FsDatasetImpl (FsVolumeList.java:addVolume(305)) - Added new volume: DS-ca6231d8-afa0-43d8-b9e6-aa2975738c0f
2018-10-18 01:09:41,323 INFO impl.FsDatasetImpl (FsDatasetImpl.java:addVolume(450)) - Added volume - [DISK]file:/grid/0/hadoop/hdfs/data, StorageType: DISK
2018-10-18 01:09:41,328 INFO impl.FsDatasetImpl (FsDatasetImpl.java:registerMBean(2277)) - Registered FSDatasetState MBean
2018-10-18 01:09:41,338 INFO checker.ThrottledAsyncChecker (ThrottledAsyncChecker.java:schedule(137)) - Scheduling a check for /grid/0/hadoop/hdfs/data
2018-10-18 01:09:41,349 INFO checker.DatasetVolumeChecker (DatasetVolumeChecker.java:checkAllVolumes(221)) - Scheduled health check for volume /grid/0/hadoop/hdfs/data
2018-10-18 01:09:41,351 INFO impl.FsDatasetImpl (FsDatasetImpl.java:addBlockPool(2788)) - Adding block pool BP-1776487077-172.27.79.197-1539042949074
2018-10-18 01:09:41,353 INFO impl.FsDatasetImpl (FsVolumeList.java:run(408)) - Scanning block pool BP-1776487077-172.27.79.197-1539042949074 on volume /grid/0/hadoop/hdfs/data...
2018-10-18 01:09:41,361 INFO impl.FsDatasetImpl (BlockPoolSlice.java:loadDfsUsed(250)) - Cached dfsUsed found for /grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current: 4145770827
2018-10-18 01:09:41,382 INFO conf.Configuration (Configuration.java:removeUndeclaredTags(3298)) - Removed undeclared tags:
2018-10-18 01:09:41,383 INFO impl.FsDatasetImpl (FsVolumeList.java:run(413)) - Time taken to scan block pool BP-1776487077-172.27.79.197-1539042949074 on /grid/0/hadoop/hdfs/data: 30ms
2018-10-18 01:09:41,383 INFO impl.FsDatasetImpl (FsVolumeList.java:addBlockPool(439)) - Total time to scan all replicas for block pool BP-1776487077-172.27.79.197-1539042949074: 31ms
2018-10-18 01:09:41,387 INFO impl.FsDatasetImpl (FsVolumeList.java:run(198)) - Adding replicas to map for block pool BP-1776487077-172.27.79.197-1539042949074 on volume /grid/0/hadoop/hdfs/data...
2018-10-18 01:09:41,387 INFO impl.BlockPoolSlice (BlockPoolSlice.java:readReplicasFromCache(777)) - Replica Cache file: /grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/replicas doesn't exist
2018-10-18 01:09:41,593 INFO impl.FsDatasetImpl (FsVolumeList.java:run(203)) - Time to add replicas to map for block pool BP-1776487077-172.27.79.197-1539042949074 on volume /grid/0/hadoop/hdfs/data: 206ms
2018-10-18 01:09:41,593 INFO impl.FsDatasetImpl (FsVolumeList.java:getAllVolumesMap(229)) - Total time to add all replicas to map: 208ms
2018-10-18 01:09:41,624 INFO datanode.VolumeScanner (VolumeScanner.java:findNextUsableBlockIter(403)) - VolumeScanner(/grid/0/hadoop/hdfs/data, DS-ca6231d8-afa0-43d8-b9e6-aa2975738c0f): no suitable block pools found to scan. Waiting 1032375592 ms.
2018-10-18 01:09:41,634 INFO datanode.DirectoryScanner (DirectoryScanner.java:start(283)) - Periodic Directory Tree Verification scan starting at 10/18/18 4:15 AM with interval of 21600000ms
2018-10-18 01:09:41,643 INFO datanode.DataNode (BPServiceActor.java:register(764)) - Block pool BP-1776487077-172.27.79.197-1539042949074 (Datanode Uuid 3092b844-60f3-43f7-aaa7-18286872be3e) service to ctr-e138-1518143905142-510793-01-000004.hwx.site/172.27.79.197:8020 beginning handshake with NN
2018-10-18 01:09:41,643 INFO datanode.DataNode (BPServiceActor.java:register(764)) - Block pool BP-1776487077-172.27.79.197-1539042949074 (Datanode Uuid 3092b844-60f3-43f7-aaa7-18286872be3e) service to ctr-e138-1518143905142-510793-01-000005.hwx.site/172.27.87.64:8020 beginning handshake with NN
2018-10-18 01:09:41,685 INFO datanode.DataNode (BPServiceActor.java:register(783)) - Block pool Block pool BP-1776487077-172.27.79.197-1539042949074 (Datanode Uuid 3092b844-60f3-43f7-aaa7-18286872be3e) service to ctr-e138-1518143905142-510793-01-000005.hwx.site/172.27.87.64:8020 successfully registered with NN
2018-10-18 01:09:41,685 INFO datanode.DataNode (BPServiceActor.java:register(783)) - Block pool Block pool BP-1776487077-172.27.79.197-1539042949074 (Datanode Uuid 3092b844-60f3-43f7-aaa7-18286872be3e) service to ctr-e138-1518143905142-510793-01-000004.hwx.site/172.27.79.197:8020 successfully registered with NN
2018-10-18 01:09:41,686 INFO datanode.DataNode (DataNode.java:registerBlockPoolWithSecretManager(1614)) - Block token params received from NN: for block pool BP-1776487077-172.27.79.197-1539042949074 keyUpdateInterval=600 min(s), tokenLifetime=600 min(s)
2018-10-18 01:09:41,691 INFO block.BlockTokenSecretManager (BlockTokenSecretManager.java:addKeys(210)) - Setting block keys
2018-10-18 01:09:41,692 INFO block.BlockTokenSecretManager (BlockTokenSecretManager.java:addKeys(210)) - Setting block keys
2018-10-18 01:09:41,692 INFO datanode.DataNode (BPServiceActor.java:offerService(612)) - For namenode ctr-e138-1518143905142-510793-01-000005.hwx.site/172.27.87.64:8020 using BLOCKREPORT_INTERVAL of 21600000msec CACHEREPORT_INTERVAL of 10000msec Initial delay: 120000msec; heartBeatInterval=3000
2018-10-18 01:09:41,692 INFO datanode.DataNode (BPServiceActor.java:offerService(612)) - For namenode ctr-e138-1518143905142-510793-01-000004.hwx.site/172.27.79.197:8020 using BLOCKREPORT_INTERVAL of 21600000msec CACHEREPORT_INTERVAL of 10000msec Initial delay: 120000msec; heartBeatInterval=3000
2018-10-18 01:09:41,796 INFO datanode.DataNode (DataNode.java:transferBlock(2328)) - DatanodeRegistration(172.27.52.130:50010, datanodeUuid=3092b844-60f3-43f7-aaa7-18286872be3e, infoPort=50075, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-57;cid=CID-2437e1cf-89ae-460e-95e7-996a9d7bccc2;nsid=99643741;c=1539042949074) Starting thread to transfer BP-1776487077-172.27.79.197-1539042949074:blk_1073742749_1948 to 172.27.56.9:50010
2018-10-18 01:09:41,801 INFO datanode.DataNode (DataNode.java:transferBlock(2328)) - DatanodeRegistration(172.27.52.130:50010, datanodeUuid=3092b844-60f3-43f7-aaa7-18286872be3e, infoPort=50075, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-57;cid=CID-2437e1cf-89ae-460e-95e7-996a9d7bccc2;nsid=99643741;c=1539042949074) Starting thread to transfer BP-1776487077-172.27.79.197-1539042949074:blk_1073742758_1957 to 172.27.56.9:50010
2018-10-18 01:09:41,807 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling blk_1073745072_4402 replica FinalizedReplica, blk_1073745072_4402, FINALIZED
getNumBytes() = 342252
getBytesOnDisk() = 342252
getVisibleLength()= 342252
getVolume() = /grid/0/hadoop/hdfs/data
getBlockURI() = file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir12/blk_1073745072 for deletion
2018-10-18 01:09:41,809 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling blk_1073745078_4410 replica FinalizedReplica, blk_1073745078_4410, FINALIZED
getNumBytes() = 342008
getBytesOnDisk() = 342008
getVisibleLength()= 342008
getVolume() = /grid/0/hadoop/hdfs/data
getBlockURI() = file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir12/blk_1073745078 for deletion
2018-10-18 01:09:41,810 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1776487077-172.27.79.197-1539042949074 blk_1073745072_4402 URI file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir12/blk_1073745072
2018-10-18 01:09:41,811 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1776487077-172.27.79.197-1539042949074 blk_1073745078_4410 URI file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir12/blk_1073745078
2018-10-18 01:09:41,836 INFO conf.Configuration (Configuration.java:removeUndeclaredTags(3298)) - Removed undeclared tags:
2018-10-18 01:09:41,934 WARN datanode.DataNode (DataNode.java:run(2571)) - DatanodeRegistration(172.27.52.130:50010, datanodeUuid=3092b844-60f3-43f7-aaa7-18286872be3e, infoPort=50075, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-57;cid=CID-2437e1cf-89ae-460e-95e7-996a9d7bccc2;nsid=99643741;c=1539042949074):Failed to transfer BP-1776487077-172.27.79.197-1539042949074:blk_1073742749_1948 to 172.27.56.9:50010 got
java.net.SocketException: Original Exception : java.io.IOException: Broken pipe
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428)
at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493)
at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608)
at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:223)
at org.apache.hadoop.hdfs.server.datanode.FileIoProvider.transferToSocketFully(FileIoProvider.java:280)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:619)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:803)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:750)
at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:2538)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Broken pipe
... 11 more
2018-10-18 01:09:41,934 WARN datanode.DataNode (DataNode.java:run(2571)) - DatanodeRegistration(172.27.52.130:50010, datanodeUuid=3092b844-60f3-43f7-aaa7-18286872be3e, infoPort=50075, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-57;cid=CID-2437e1cf-89ae-460e-95e7-996a9d7bccc2;nsid=99643741;c=1539042949074):Failed to transfer BP-1776487077-172.27.79.197-1539042949074:blk_1073742758_1957 to 172.27.56.9:50010 got
java.net.SocketException: Original Exception : java.io.IOException: Broken pipe
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428)
at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493)
at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608)
at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:223)
at org.apache.hadoop.hdfs.server.datanode.FileIoProvider.transferToSocketFully(FileIoProvider.java:280)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:619)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:803)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:750)
at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:2538)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Broken pipe
... 11 more
2018-10-18 01:09:41,978 INFO impl.RaftServerProxy (ConfUtils.java:logGet(43)) - raft.rpc.type = GRPC (default)
2018-10-18 01:09:41,995 INFO grpc.GrpcConfigKeys$Server (ConfUtils.java:logGet(43)) - raft.grpc.server.port = 9858 (custom)
2018-10-18 01:09:41,996 INFO server.GrpcService (ConfUtils.java:logGet(43)) - raft.grpc.message.size.max = 167772160 (custom)
2018-10-18 01:09:41,999 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.log.appender.buffer.capacity = 134217728 (custom)
2018-10-18 01:09:42,000 INFO server.GrpcService (ConfUtils.java:logGet(43)) - raft.grpc.flow.control.window = 1MB (=1048576) (default)
2018-10-18 01:09:42,001 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.rpc.request.timeout = 3000 ms (custom)
2018-10-18 01:09:42,087 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.storage.dir = [/tmp/data/disk1/ozone/ratis] (custom)
2018-10-18 01:09:42,091 INFO impl.RaftServerProxy (RaftServerProxy.java:lambda$null$0(179)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: found a subdirectory /tmp/data/disk1/ozone/ratis/ba0b9d2c-ba94-443d-a784-80012dd9fe74
2018-10-18 01:09:42,097 INFO impl.RaftServerProxy (RaftServerProxy.java:addNew(85)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: addNew group-80012DD9FE74:[] returns group-80012DD9FE74:java.util.concurrent.CompletableFuture@1e6dad8[Not completed]
2018-10-18 01:09:42,102 INFO replication.SimpleContainerDownloader (SimpleContainerDownloader.java:<init>(73)) - Starting container downloader service to copy containers to replicate.
2018-10-18 01:09:42,131 INFO impl.RaftServerImpl (RaftServerImpl.java:<init>(82)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: new RaftServerImpl for group-80012DD9FE74:[] with ContainerStateMachine:uninitialized
2018-10-18 01:09:42,133 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.rpc.timeout.min = 1 s (custom)
2018-10-18 01:09:42,134 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.rpc.timeout.max = 1200 ms (custom)
2018-10-18 01:09:42,134 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.rpcslowness.timeout = 120 s (custom)
2018-10-18 01:09:42,143 INFO impl.RaftServerImpl (ServerState.java:<init>(98)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: ConfigurationManager, init=-1: [], old=null, confs=<EMPTY_MAP>
2018-10-18 01:09:42,144 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.storage.dir = [/tmp/data/disk1/ozone/ratis] (custom)
2018-10-18 01:09:42,152 INFO storage.RaftStorageDirectory (RaftStorageDirectory.java:tryLock(316)) - Lock on /tmp/data/disk1/ozone/ratis/ba0b9d2c-ba94-443d-a784-80012dd9fe74/in_use.lock acquired by nodename 3596688@ctr-e138-1518143905142-510793-01-000007.hwx.site
2018-10-18 01:09:42,160 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.leader.election.timeout = 120 s (custom)
2018-10-18 01:09:42,164 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.log.use.memory = false (default)
2018-10-18 01:09:42,166 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.log.appender.buffer.capacity = 134217728 (custom)
2018-10-18 01:09:42,167 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.log.segment.size.max = 1073741824 (custom)
2018-10-18 01:09:42,171 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.log.segment.cache.num.max = 2 (custom)
2018-10-18 01:09:42,177 INFO storage.RaftLogWorker (RaftLogWorker.java:<init>(87)) - new 69d15283-4e2e-4c30-a028-f2bad0f83cc1-RaftLogWorker for Storage Directory /tmp/data/disk1/ozone/ratis/ba0b9d2c-ba94-443d-a784-80012dd9fe74
2018-10-18 01:09:42,178 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.log.segment.size.max = 1073741824 (custom)
2018-10-18 01:09:42,179 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.log.preallocated.size = 134217728 (custom)
2018-10-18 01:09:42,179 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.log.write.buffer.size = 33554432 (custom)
2018-10-18 01:09:42,180 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.log.force.sync.num = 128 (default)
2018-10-18 01:09:42,236 INFO impl.RaftServerImpl (ServerState.java:setRaftConf(350)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: set configuration 0: [41636d6e-31cc-419b-bd8d-1a7e5e7d7edd:172.27.23.193:9858, 69d15283-4e2e-4c30-a028-f2bad0f83cc1:172.27.52.130:9858, a698a509-0816-4719-9a0f-36d1b503c249:172.27.17.133:9858], old=null at 0
2018-10-18 01:09:42,736 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.snapshot.auto.trigger.enabled = false (default)
2018-10-18 01:09:42,738 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.snapshot.auto.trigger.threshold = 400000 (default)
2018-10-18 01:09:42,739 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.retrycache.capacity = 4096 (default)
2018-10-18 01:09:42,739 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.retrycache.expirytime = 600000 ms (custom)
2018-10-18 01:09:43,342 INFO netty.ObjectStoreRestHttpServer (ObjectStoreRestHttpServer.java:start(148)) - Listening HDDS REST traffic on /0.0.0.0:9880
2018-10-18 01:09:43,343 INFO ozone.HddsDatanodeService (HddsDatanodeService.java:startPlugins(158)) - Started plug-in org.apache.hadoop.ozone.web.OzoneHddsDatanodeService@70d63e05
2018-10-18 01:09:43,344 INFO datanode.DataNode (DataNode.java:startPlugins(981)) - Started plug-in org.apache.hadoop.ozone.HddsDatanodeService@43984213
2018-10-18 01:09:43,348 INFO ozoneimpl.OzoneContainer (OzoneContainer.java:start(137)) - Attempting to start container services.
2018-10-18 01:09:43,348 INFO ratis.XceiverServerRatis (XceiverServerRatis.java:start(291)) - Starting XceiverServerRatis 69d15283-4e2e-4c30-a028-f2bad0f83cc1 at port 9858
2018-10-18 01:09:43,371 INFO impl.RaftServerImpl (RaftServerImpl.java:start(163)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: start group-80012DD9FE74
2018-10-18 01:09:43,375 INFO impl.RaftServerImpl (RaftServerImpl.java:setRole(154)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1 changes role from null to FOLLOWER at term 2 for startAsFollower
2018-10-18 01:09:43,377 INFO impl.RoleInfo (RoleInfo.java:updateAndGet(139)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: start FollowerState
2018-10-18 01:09:43,380 INFO util.JmxRegister (JmxRegister.java:tryRegister(44)) - Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-80012DD9FE74,id=69d15283-4e2e-4c30-a028-f2bad0f83cc1
2018-10-18 01:09:43,380 INFO impl.RaftServerProxy (RaftServerProxy.java:lambda$start$3(282)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: start RPC server
2018-10-18 01:09:43,719 INFO server.GrpcService (GrpcService.java:startImpl(107)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: GrpcService started, listening on 0.0.0.0/0.0.0.0:9858
2018-10-18 01:09:44,536 INFO impl.FollowerState (FollowerState.java:run(76)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1 changes to CANDIDATE, lastRpcTime:1159, electionTimeout:1158ms
2018-10-18 01:09:44,537 INFO impl.RoleInfo (RoleInfo.java:shutdownFollowerState(117)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: shutdown FollowerState
2018-10-18 01:09:44,538 INFO impl.RaftServerImpl (RaftServerImpl.java:setRole(154)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1 changes role from FOLLOWER to CANDIDATE at term 2 for changeToCandidate
2018-10-18 01:09:44,541 INFO impl.RoleInfo (RoleInfo.java:updateAndGet(139)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: start LeaderElection
2018-10-18 01:09:44,545 INFO impl.LeaderElection (LeaderElection.java:askForVotes(127)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: begin an election in Term 3
2018-10-18 01:09:44,643 INFO impl.RaftServerImpl (RaftServerImpl.java:setRole(154)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1 changes role from CANDIDATE to FOLLOWER at term 6 for changeToFollower
2018-10-18 01:09:44,643 INFO impl.RoleInfo (RoleInfo.java:shutdownLeaderElection(130)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: shutdown LeaderElection
2018-10-18 01:09:44,643 INFO impl.RoleInfo (RoleInfo.java:updateAndGet(139)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: start FollowerState
2018-10-18 01:09:44,699 INFO datanode.DataNode (DataNode.java:transferBlock(2328)) - DatanodeRegistration(172.27.52.130:50010, datanodeUuid=3092b844-60f3-43f7-aaa7-18286872be3e, infoPort=50075, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-57;cid=CID-2437e1cf-89ae-460e-95e7-996a9d7bccc2;nsid=99643741;c=1539042949074) Starting thread to transfer BP-1776487077-172.27.79.197-1539042949074:blk_1073742683_1882 to 172.27.56.9:50010
2018-10-18 01:09:44,699 INFO datanode.DataNode (DataNode.java:transferBlock(2328)) - DatanodeRegistration(172.27.52.130:50010, datanodeUuid=3092b844-60f3-43f7-aaa7-18286872be3e, infoPort=50075, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-57;cid=CID-2437e1cf-89ae-460e-95e7-996a9d7bccc2;nsid=99643741;c=1539042949074) Starting thread to transfer BP-1776487077-172.27.79.197-1539042949074:blk_1073742749_1948 to 172.27.56.9:50010
2018-10-18 01:09:44,719 WARN datanode.DataNode (DataNode.java:run(2571)) - DatanodeRegistration(172.27.52.130:50010, datanodeUuid=3092b844-60f3-43f7-aaa7-18286872be3e, infoPort=50075, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-57;cid=CID-2437e1cf-89ae-460e-95e7-996a9d7bccc2;nsid=99643741;c=1539042949074):Failed to transfer BP-1776487077-172.27.79.197-1539042949074:blk_1073742683_1882 to 172.27.56.9:50010 got
java.net.SocketException: Original Exception : java.io.IOException: Connection reset by peer
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428)
at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493)
at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608)
at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:223)
at org.apache.hadoop.hdfs.server.datanode.FileIoProvider.transferToSocketFully(FileIoProvider.java:280)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:619)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:803)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:750)
at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:2538)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Connection reset by peer
... 11 more
2018-10-18 01:09:44,721 WARN datanode.DataNode (DataNode.java:run(2571)) - DatanodeRegistration(172.27.52.130:50010, datanodeUuid=3092b844-60f3-43f7-aaa7-18286872be3e, infoPort=50075, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-57;cid=CID-2437e1cf-89ae-460e-95e7-996a9d7bccc2;nsid=99643741;c=1539042949074):Failed to transfer BP-1776487077-172.27.79.197-1539042949074:blk_1073742749_1948 to 172.27.56.9:50010 got
java.net.SocketException: Original Exception : java.io.IOException: Connection reset by peer
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428)
at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493)
at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608)
at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:223)
at org.apache.hadoop.hdfs.server.datanode.FileIoProvider.transferToSocketFully(FileIoProvider.java:280)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:619)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:803)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:750)
at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:2538)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Connection reset by peer
... 11 more
2018-10-18 01:09:44,773 INFO impl.LeaderElection (LeaderElection.java:waitForResults(230)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1 got exception when requesting votes: {}
java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:214)
at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:146)
at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:102)
Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:222)
at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:203)
at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:132)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265)
at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:61)
at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:150)
at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$0(LeaderElection.java:188)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.27.17.133:9858
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:325)
at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:634)
at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)
at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 more
Caused by: java.net.ConnectException: Connection refused
... 11 more
2018-10-18 01:09:44,776 INFO impl.LeaderElection (LeaderElection.java:logAndReturn(46)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Election REJECTED; received 0 response(s) [] and 1 exception(s); 69d15283-4e2e-4c30-a028-f2bad0f83cc1:t6, leader=null, voted=null, raftlog=[(t:2, i:12)], conf=0: [41636d6e-31cc-419b-bd8d-1a7e5e7d7edd:172.27.23.193:9858, 69d15283-4e2e-4c30-a028-f2bad0f83cc1:172.27.52.130:9858, a698a509-0816-4719-9a0f-36d1b503c249:172.27.17.133:9858], old=null
2018-10-18 01:09:44,776 INFO impl.LeaderElection (LeaderElection.java:logAndReturn(52)) - 0: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2018-10-18 01:09:45,870 INFO impl.RaftServerImpl (ServerState.java:setLeader(260)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: change Leader from null to 41636d6e-31cc-419b-bd8d-1a7e5e7d7edd at term 7 for appendEntries, leader elected after 3710ms
2018-10-18 01:09:45,873 INFO storage.RaftLogWorker (RaftLogWorker.java:rollLogSegment(258)) - Rolling segment:69d15283-4e2e-4c30-a028-f2bad0f83cc1-RaftLogWorker index to:13
2018-10-18 01:09:45,878 INFO impl.RaftServerImpl (ServerState.java:setRaftConf(350)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: set configuration 13: [41636d6e-31cc-419b-bd8d-1a7e5e7d7edd:172.27.23.193:9858, 69d15283-4e2e-4c30-a028-f2bad0f83cc1:172.27.52.130:9858, a698a509-0816-4719-9a0f-36d1b503c249:172.27.17.133:9858], old=null at 13
2018-10-18 01:09:46,763 WARN keyvalue.KeyValueHandler (KeyValueHandler.java:handleCreateContainer(248)) - Container already exists.container Id 4
2018-10-18 01:09:46,811 WARN impl.ChunkManagerImpl (ChunkUtils.java:validateChunkForOverwrite(244)) - Duplicate write chunk request. Chunk overwrite without explicit request. ChunkInfo{chunkName='d0f5b669c4af90a4e22ad7443f4a715b_stream_f508e2e3-c758-4064-8842-2bfdd68ea235_chunk_1, offset=0, len=9}
2018-10-18 01:09:46,812 WARN impl.ChunkManagerImpl (ChunkManagerImpl.java:writeChunk(130)) - ChunkFile already exists/grid/0/hadoop/hdfs/data/hdds/3be4032c-ab7b-4e18-9d1c-3a6c7ba2c020/current/containerDir0/4/chunks/d0f5b669c4af90a4e22ad7443f4a715b_stream_f508e2e3-c758-4064-8842-2bfdd68ea235_chunk_1
2018-10-18 01:09:46,821 WARN keyvalue.KeyValueHandler (KeyValueHandler.java:handleCreateContainer(248)) - Container already exists.container Id 6
2018-10-18 01:09:46,821 WARN impl.BlockManagerImpl (BlockManagerImpl.java:putBlock(107)) - blockCommitSequenceId 3 in the Container Db is greater than the supplied value 3 .Ignoring it
2018-10-18 01:09:46,822 WARN impl.ChunkManagerImpl (ChunkUtils.java:validateChunkForOverwrite(244)) - Duplicate write chunk request. Chunk overwrite without explicit request. ChunkInfo{chunkName='4846b122487a4f1812228e0ba325d76b_stream_d6622d50-81bc-442a-9e54-a5bd32d9ce2a_chunk_1, offset=0, len=73}
2018-10-18 01:09:46,822 WARN keyvalue.KeyValueHandler (KeyValueHandler.java:handleCreateContainer(248)) - Container already exists.container Id 8
2018-10-18 01:09:46,823 WARN impl.ChunkManagerImpl (ChunkManagerImpl.java:writeChunk(130)) - ChunkFile already exists/grid/0/hadoop/hdfs/data/hdds/3be4032c-ab7b-4e18-9d1c-3a6c7ba2c020/current/containerDir0/6/chunks/4846b122487a4f1812228e0ba325d76b_stream_d6622d50-81bc-442a-9e54-a5bd32d9ce2a_chunk_1
2018-10-18 01:09:46,823 WARN impl.ChunkManagerImpl (ChunkUtils.java:validateChunkForOverwrite(244)) - Duplicate write chunk request. Chunk overwrite without explicit request. ChunkInfo{chunkName='03620139de43df179385cd5187a534bf_stream_f2f84ff7-fcfe-4b50-a2dc-99ae2c3058cd_chunk_1, offset=0, len=9}
2018-10-18 01:09:46,823 WARN impl.BlockManagerImpl (BlockManagerImpl.java:putBlock(107)) - blockCommitSequenceId 6 in the Container Db is greater than the supplied value 6 .Ignoring it
2018-10-18 01:09:46,824 WARN keyvalue.KeyValueHandler (KeyValueHandler.java:handleCreateContainer(248)) - Container already exists.container Id 10
2018-10-18 01:09:46,823 WARN impl.ChunkManagerImpl (ChunkManagerImpl.java:writeChunk(130)) - ChunkFile already exists/grid/0/hadoop/hdfs/data/hdds/3be4032c-ab7b-4e18-9d1c-3a6c7ba2c020/current/containerDir0/8/chunks/03620139de43df179385cd5187a534bf_stream_f2f84ff7-fcfe-4b50-a2dc-99ae2c3058cd_chunk_1
2018-10-18 01:09:46,826 WARN impl.ChunkManagerImpl (ChunkUtils.java:validateChunkForOverwrite(244)) - Duplicate write chunk request. Chunk overwrite without explicit request. ChunkInfo{chunkName='84a154f61dd461991a72934f883fe9d7_stream_d75e58dd-0bb6-4bc1-94b8-b901cd910688_chunk_1, offset=0, len=73}
2018-10-18 01:09:46,826 WARN impl.BlockManagerImpl (BlockManagerImpl.java:putBlock(107)) - blockCommitSequenceId 9 in the Container Db is greater than the supplied value 9 .Ignoring it
2018-10-18 01:09:46,826 WARN impl.ChunkManagerImpl (ChunkManagerImpl.java:writeChunk(130)) - ChunkFile already exists/grid/0/hadoop/hdfs/data/hdds/3be4032c-ab7b-4e18-9d1c-3a6c7ba2c020/current/containerDir0/10/chunks/84a154f61dd461991a72934f883fe9d7_stream_d75e58dd-0bb6-4bc1-94b8-b901cd910688_chunk_1
2018-10-18 01:09:46,827 WARN impl.BlockManagerImpl (BlockManagerImpl.java:putBlock(107)) - blockCommitSequenceId 12 in the Container Db is greater than the supplied value 12 .Ignoring it
2018-10-18 01:09:47,982 INFO impl.RaftServerProxy (RaftServerProxy.java:remove(91)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: remove FOLLOWER group-80012DD9FE74 69d15283-4e2e-4c30-a028-f2bad0f83cc1:t7, leader=41636d6e-31cc-419b-bd8d-1a7e5e7d7edd, voted=41636d6e-31cc-419b-bd8d-1a7e5e7d7edd, raftlog=[(t:7, i:17)], conf=13: [41636d6e-31cc-419b-bd8d-1a7e5e7d7edd:172.27.23.193:9858, 69d15283-4e2e-4c30-a028-f2bad0f83cc1:172.27.52.130:9858, a698a509-0816-4719-9a0f-36d1b503c249:172.27.17.133:9858], old=null RUNNING
2018-10-18 01:09:47,985 INFO impl.RaftServerImpl (RaftServerImpl.java:lambda$shutdown$3(228)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: shutdown group-80012DD9FE74
2018-10-18 01:09:47,985 INFO util.JmxRegister (JmxRegister.java:unregister(73)) - Successfully un-registered JMX Bean with object name Ratis:service=RaftServer,group=group-80012DD9FE74,id=69d15283-4e2e-4c30-a028-f2bad0f83cc1
2018-10-18 01:09:47,986 INFO impl.RoleInfo (RoleInfo.java:shutdownFollowerState(117)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: shutdown FollowerState
2018-10-18 01:09:47,987 INFO impl.StateMachineUpdater (StateMachineUpdater.java:run(171)) - StateMachineUpdater-69d15283-4e2e-4c30-a028-f2bad0f83cc1: the StateMachineUpdater is interrupted and will exit.
2018-10-18 01:09:47,987 INFO impl.FollowerState (FollowerState.java:run(84)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: FollowerState was interrupted: java.lang.InterruptedException: sleep interrupted
2018-10-18 01:09:47,987 INFO impl.RaftServerImpl (ServerState.java:close(381)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1 closes. The last applied log index is 17
2018-10-18 01:09:47,990 INFO storage.RaftLogWorker (RaftLogWorker.java:run(196)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1-RaftLogWorker was interrupted, exiting. There are 0 tasks remaining in the queue.
2018-10-18 01:09:48,009 INFO server.GrpcServerProtocolService (GrpcServerProtocolService.java:onCompleted(100)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: appendEntries completed
2018-10-18 01:09:48,018 INFO storage.RaftLogWorker (RaftLogWorker.java:close(127)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1-RaftLogWorker close()
2018-10-18 01:09:50,698 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling blk_1073744642_3966 replica FinalizedReplica, blk_1073744642_3966, FINALIZED
getNumBytes() = 1719
getBytesOnDisk() = 1719
getVisibleLength()= 1719
getVolume() = /grid/0/hadoop/hdfs/data
getBlockURI() = file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir11/blk_1073744642 for deletion
2018-10-18 01:09:50,699 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling blk_1073742050_1226 replica FinalizedReplica, blk_1073742050_1226, FINALIZED
getNumBytes() = 41413541
getBytesOnDisk() = 41413541
getVisibleLength()= 41413541
getVolume() = /grid/0/hadoop/hdfs/data
getBlockURI() = file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir0/blk_1073742050 for deletion
2018-10-18 01:09:50,699 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling blk_1073742244_1437 replica FinalizedReplica, blk_1073742244_1437, FINALIZED
getNumBytes() = 3317961
getBytesOnDisk() = 3317961
getVisibleLength()= 3317961
getVolume() = /grid/0/hadoop/hdfs/data
getBlockURI() = file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir1/blk_1073742244 for deletion
2018-10-18 01:09:50,700 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling blk_1073743429_2652 replica FinalizedReplica, blk_1073743429_2652, FINALIZED
getNumBytes() = 30
getBytesOnDisk() = 30
getVisibleLength()= 30
getVolume() = /grid/0/hadoop/hdfs/data
getBlockURI() = file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir6/blk_1073743429 for deletion
2018-10-18 01:09:50,700 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling blk_1073741864_1040 replica FinalizedReplica, blk_1073741864_1040, FINALIZED
getNumBytes() = 134217728
getBytesOnDisk() = 134217728
getVisibleLength()= 134217728
getVolume() = /grid/0/hadoop/hdfs/data
getBlockURI() = file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir0/blk_1073741864 for deletion
2018-10-18 01:09:50,700 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling blk_1073742824_2023 replica FinalizedReplica, blk_1073742824_2023, FINALIZED
getNumBytes() = 18391
getBytesOnDisk() = 18391
getVisibleLength()= 18391
getVolume() = /grid/0/hadoop/hdfs/data
getBlockURI() = file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir3/blk_1073742824 for deletion
2018-10-18 01:09:50,701 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling blk_1073744232_3540 replica FinalizedReplica, blk_1073744232_3540, FINALIZED
getNumBytes() = 100851
getBytesOnDisk() = 100851
getVisibleLength()= 100851
getVolume() = /grid/0/hadoop/hdfs/data
getBlockURI() = file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir9/blk_1073744232 for deletion
2018-10-18 01:09:50,701 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling blk_1073744557_3881 replica FinalizedReplica, blk_1073744557_3881, FINALIZED
getNumBytes() = 15092
getBytesOnDisk() = 15092
getVisibleLength()= 15092
getVolume() = /grid/0/hadoop/hdfs/data
getBlockURI() = file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir10/blk_1073744557 for deletion
2018-10-18 01:09:50,702 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling blk_1073744527_3851 replica FinalizedReplica, blk_1073744527_3851, FINALIZED
getNumBytes() = 269070
getBytesOnDisk() = 269070
getVisibleLength()= 269070
getVolume() = /grid/0/hadoop/hdfs/data
getBlockURI() = file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir10/blk_1073744527 for deletion
2018-10-18 01:09:50,702 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1776487077-172.27.79.197-1539042949074 blk_1073744642_3966 URI file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir11/blk_1073744642
2018-10-18 01:09:50,702 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling blk_1073744239_3582 replica FinalizedReplica, blk_1073744239_3582, FINALIZED
getNumBytes() = 100311
getBytesOnDisk() = 100311
getVisibleLength()= 100311
getVolume() = /grid/0/hadoop/hdfs/data
getBlockURI() = file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir9/blk_1073744239 for deletion
2018-10-18 01:09:50,703 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling blk_1073744976_4306 replica FinalizedReplica, blk_1073744976_4306, FINALIZED
getNumBytes() = 20411938
getBytesOnDisk() = 20411938
getVisibleLength()= 20411938
getVolume() = /grid/0/hadoop/hdfs/data
getBlockURI() = file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir12/blk_1073744976 for deletion
2018-10-18 01:09:50,703 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling blk_1073743377_2600 replica FinalizedReplica, blk_1073743377_2600, FINALIZED
getNumBytes() = 30
getBytesOnDisk() = 30
getVisibleLength()= 30
getVolume() = /grid/0/hadoop/hdfs/data
getBlockURI() = file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir6/blk_1073743377 for deletion
2018-10-18 01:09:50,704 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling blk_1073741877_1053 replica FinalizedReplica, blk_1073741877_1053, FINALIZED
getNumBytes() = 411
getBytesOnDisk() = 411
getVisibleLength()= 411
getVolume() = /grid/0/hadoop/hdfs/data
getBlockURI() = file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir0/blk_1073741877 for deletion
2018-10-18 01:09:50,704 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling blk_1073742903_2126 replica FinalizedReplica, blk_1073742903_2126, FINALIZED
getNumBytes() = 9601
getBytesOnDisk() = 9601
getVisibleLength()= 9601
getVolume() = /grid/0/hadoop/hdfs/data
getBlockURI() = file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir4/blk_1073742903 for deletion
2018-10-18 01:09:50,705 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling blk_1073744535_3859 replica FinalizedReplica, blk_1073744535_3859, FINALIZED
getNumBytes() = 20406232
getBytesOnDisk() = 20406232
getVisibleLength()= 20406232
getVolume() = /grid/0/hadoop/hdfs/data
getBlockURI() = file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir10/blk_1073744535 for deletion
2018-10-18 01:09:50,705 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling blk_1073741883_1059 replica FinalizedReplica, blk_1073741883_1059, FINALIZED
getNumBytes() = 6633
getBytesOnDisk() = 6633
getVisibleLength()= 6633
getVolume() = /grid/0/hadoop/hdfs/data
getBlockURI() = file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir0/blk_1073741883 for deletion
2018-10-18 01:09:50,706 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling blk_1073742910_2133 replica FinalizedReplica, blk_1073742910_2133, FINALIZED
getNumBytes() = 30
getBytesOnDisk() = 30
getVisibleLength()= 30
getVolume() = /grid/0/hadoop/hdfs/data
getBlockURI() = file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir4/blk_1073742910 for deletion
2018-10-18 01:09:50,706 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling blk_1073742302_1495 replica FinalizedReplica, blk_1073742302_1495, FINALIZED
getNumBytes() = 215755
getBytesOnDisk() = 215755
getVisibleLength()= 215755
getVolume() = /grid/0/hadoop/hdfs/data
getBlockURI() = file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir1/blk_1073742302 for deletion
2018-10-18 01:09:50,710 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1776487077-172.27.79.197-1539042949074 blk_1073742050_1226 URI file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir0/blk_1073742050
2018-10-18 01:09:50,711 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1776487077-172.27.79.197-1539042949074 blk_1073742244_1437 URI file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir1/blk_1073742244
2018-10-18 01:09:50,711 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1776487077-172.27.79.197-1539042949074 blk_1073743429_2652 URI file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir6/blk_1073743429
2018-10-18 01:09:50,720 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1776487077-172.27.79.197-1539042949074 blk_1073741864_1040 URI file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir0/blk_1073741864
2018-10-18 01:09:50,720 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1776487077-172.27.79.197-1539042949074 blk_1073742824_2023 URI file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir3/blk_1073742824
2018-10-18 01:09:50,720 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1776487077-172.27.79.197-1539042949074 blk_1073744232_3540 URI file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir9/blk_1073744232
2018-10-18 01:09:50,721 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1776487077-172.27.79.197-1539042949074 blk_1073744557_3881 URI file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir10/blk_1073744557
2018-10-18 01:09:50,721 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1776487077-172.27.79.197-1539042949074 blk_1073744527_3851 URI file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir10/blk_1073744527
2018-10-18 01:09:50,722 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1776487077-172.27.79.197-1539042949074 blk_1073744239_3582 URI file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir9/blk_1073744239
2018-10-18 01:09:50,727 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1776487077-172.27.79.197-1539042949074 blk_1073744976_4306 URI file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir12/blk_1073744976
2018-10-18 01:09:50,727 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1776487077-172.27.79.197-1539042949074 blk_1073743377_2600 URI file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir6/blk_1073743377
2018-10-18 01:09:50,728 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1776487077-172.27.79.197-1539042949074 blk_1073741877_1053 URI file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir0/blk_1073741877
2018-10-18 01:09:50,728 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1776487077-172.27.79.197-1539042949074 blk_1073742903_2126 URI file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir4/blk_1073742903
2018-10-18 01:09:50,732 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1776487077-172.27.79.197-1539042949074 blk_1073744535_3859 URI file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir10/blk_1073744535
2018-10-18 01:09:50,732 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1776487077-172.27.79.197-1539042949074 blk_1073741883_1059 URI file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir0/blk_1073741883
2018-10-18 01:09:50,733 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1776487077-172.27.79.197-1539042949074 blk_1073742910_2133 URI file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir4/blk_1073742910
2018-10-18 01:09:50,733 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1776487077-172.27.79.197-1539042949074 blk_1073742302_1495 URI file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/finalized/subdir0/subdir1/blk_1073742302
2018-10-18 01:09:54,518 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:09:55,674 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:09:56,772 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:09:57,835 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:09:59,028 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:00,246 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:01,374 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:02,440 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:03,519 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:04,619 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:05,815 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:07,024 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:08,142 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:09,213 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:10,340 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:11,564 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:12,740 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:13,888 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:14,555 INFO datanode.DataNode (DataXceiver.java:writeBlock(738)) - Receiving BP-1776487077-172.27.79.197-1539042949074:blk_1073745089_4437 src: /172.27.17.133:59634 dest: /172.27.52.130:50010
2018-10-18 01:10:14,916 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:16,136 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:17,181 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:18,211 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:19,440 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:20,567 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:20,730 INFO commandhandler.DeleteBlocksCommandHandler (DeleteBlocksCommandHandler.java:handle(106)) - Start to delete container blocks, TXIDs=[2(0),3(0)], numOfContainers=2, numOfBlocks=2
2018-10-18 01:10:21,586 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:22,730 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:25,198 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:26,346 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:26,726 INFO datanode.DataNode (BPServiceActor.java:blockReport(422)) - Successfully sent block report 0x1a9a287ee94a56d6, containing 1 storage report(s), of which we sent 1. The reports had 662 total blocks and used 1 RPC(s). This took 10 msec to generate and 15 msecs for RPC and NN processing. Got back one command: FinalizeCommand/5.
2018-10-18 01:10:26,727 INFO datanode.DataNode (BPOfferService.java:processCommandFromActive(759)) - Got finalize command for block pool BP-1776487077-172.27.79.197-1539042949074
2018-10-18 01:10:27,522 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:28,704 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:29,791 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:30,981 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:32,020 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:33,114 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:34,296 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:35,403 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:36,467 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:37,640 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:38,730 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:39,778 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:40,922 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:41,596 INFO background.BlockDeletingService (BlockDeletingService.java:getTasks(123)) - Plan to choose 10 containers for block deletion, actually returns 2 valid containers.
2018-10-18 01:10:41,609 INFO background.BlockDeletingService (BlockDeletingService.java:call(245)) - Container: 6, deleted blocks: 1, task elapsed time: 9ms
2018-10-18 01:10:41,609 INFO background.BlockDeletingService (BlockDeletingService.java:call(245)) - Container: 10, deleted blocks: 1, task elapsed time: 9ms
2018-10-18 01:10:42,021 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:43,101 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:44,164 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:45,280 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:46,383 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:47,459 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:48,554 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:49,732 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:50,750 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:51,791 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:52,331 INFO impl.RaftServerProxy (RaftServerProxy.java:addNew(85)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: addNew group-5382A2B56560:[41636d6e-31cc-419b-bd8d-1a7e5e7d7edd:172.27.23.193:9858, 69d15283-4e2e-4c30-a028-f2bad0f83cc1:172.27.52.130:9858, a698a509-0816-4719-9a0f-36d1b503c249:172.27.17.133:9858] returns group-5382A2B56560:java.util.concurrent.CompletableFuture@2a185879[Not completed]
2018-10-18 01:10:52,331 INFO impl.RaftServerImpl (RaftServerImpl.java:<init>(82)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: new RaftServerImpl for group-5382A2B56560:[41636d6e-31cc-419b-bd8d-1a7e5e7d7edd:172.27.23.193:9858, 69d15283-4e2e-4c30-a028-f2bad0f83cc1:172.27.52.130:9858, a698a509-0816-4719-9a0f-36d1b503c249:172.27.17.133:9858] with ContainerStateMachine:69d15283-4e2e-4c30-a028-f2bad0f83cc1:group-80012DD9FE74
2018-10-18 01:10:52,332 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.rpc.timeout.min = 1 s (custom)
2018-10-18 01:10:52,332 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.rpc.timeout.max = 1200 ms (custom)
2018-10-18 01:10:52,332 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.rpcslowness.timeout = 120 s (custom)
2018-10-18 01:10:52,333 INFO impl.RaftServerImpl (ServerState.java:<init>(98)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: ConfigurationManager, init=-1: [41636d6e-31cc-419b-bd8d-1a7e5e7d7edd:172.27.23.193:9858, 69d15283-4e2e-4c30-a028-f2bad0f83cc1:172.27.52.130:9858, a698a509-0816-4719-9a0f-36d1b503c249:172.27.17.133:9858], old=null, confs=<EMPTY_MAP>
2018-10-18 01:10:52,333 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.storage.dir = [/tmp/data/disk1/ozone/ratis] (custom)
2018-10-18 01:10:52,338 INFO storage.RaftStorageDirectory (RaftStorageDirectory.java:analyzeStorage(234)) - The storage directory /tmp/data/disk1/ozone/ratis/bc3e8eb4-a680-40fb-b85b-5382a2b56560 does not exist. Creating ...
2018-10-18 01:10:52,341 INFO storage.RaftStorageDirectory (RaftStorageDirectory.java:tryLock(316)) - Lock on /tmp/data/disk1/ozone/ratis/bc3e8eb4-a680-40fb-b85b-5382a2b56560/in_use.lock acquired by nodename 3596688@ctr-e138-1518143905142-510793-01-000007.hwx.site
2018-10-18 01:10:52,342 INFO storage.RaftStorage (RaftStorage.java:format(70)) - Storage directory /tmp/data/disk1/ozone/ratis/bc3e8eb4-a680-40fb-b85b-5382a2b56560 has been successfully formatted.
2018-10-18 01:10:52,343 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.leader.election.timeout = 120 s (custom)
2018-10-18 01:10:52,343 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.log.use.memory = false (default)
2018-10-18 01:10:52,343 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.log.appender.buffer.capacity = 134217728 (custom)
2018-10-18 01:10:52,344 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.log.segment.size.max = 1073741824 (custom)
2018-10-18 01:10:52,344 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.log.segment.cache.num.max = 2 (custom)
2018-10-18 01:10:52,344 INFO storage.RaftLogWorker (RaftLogWorker.java:<init>(87)) - new 69d15283-4e2e-4c30-a028-f2bad0f83cc1-RaftLogWorker for Storage Directory /tmp/data/disk1/ozone/ratis/bc3e8eb4-a680-40fb-b85b-5382a2b56560
2018-10-18 01:10:52,344 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.log.segment.size.max = 1073741824 (custom)
2018-10-18 01:10:52,344 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.log.preallocated.size = 134217728 (custom)
2018-10-18 01:10:52,344 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.log.write.buffer.size = 33554432 (custom)
2018-10-18 01:10:52,345 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.log.force.sync.num = 128 (default)
2018-10-18 01:10:52,345 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.snapshot.auto.trigger.enabled = false (default)
2018-10-18 01:10:52,346 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.snapshot.auto.trigger.threshold = 400000 (default)
2018-10-18 01:10:52,346 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.retrycache.capacity = 4096 (default)
2018-10-18 01:10:52,346 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.retrycache.expirytime = 600000 ms (custom)
2018-10-18 01:10:52,347 INFO impl.RaftServerImpl (RaftServerImpl.java:start(163)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: start group-5382A2B56560
2018-10-18 01:10:52,347 INFO impl.RaftServerImpl (RaftServerImpl.java:setRole(154)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1 changes role from null to FOLLOWER at term 0 for startAsFollower
2018-10-18 01:10:52,347 INFO impl.RoleInfo (RoleInfo.java:updateAndGet(139)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: start FollowerState
2018-10-18 01:10:52,348 INFO util.JmxRegister (JmxRegister.java:tryRegister(44)) - Successfully registered JMX Bean with object name Ratis:service=RaftServer,group=group-5382A2B56560,id=69d15283-4e2e-4c30-a028-f2bad0f83cc1
2018-10-18 01:10:52,908 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:53,429 INFO impl.RaftServerImpl (ServerState.java:setLeader(260)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: change Leader from null to 41636d6e-31cc-419b-bd8d-1a7e5e7d7edd at term 1 for appendEntries, leader elected after 1086ms
2018-10-18 01:10:53,430 INFO impl.RaftServerImpl (ServerState.java:setRaftConf(350)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: set configuration 0: [41636d6e-31cc-419b-bd8d-1a7e5e7d7edd:172.27.23.193:9858, 69d15283-4e2e-4c30-a028-f2bad0f83cc1:172.27.52.130:9858, a698a509-0816-4719-9a0f-36d1b503c249:172.27.17.133:9858], old=null at 0
2018-10-18 01:10:53,927 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:54,216 WARN client.GrpcClientProtocolService (LogUtils.java:warn(135)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1-0: onError: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2018-10-18 01:10:54,959 WARN server.GrpcServerProtocolService (LogUtils.java:warn(135)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: appendEntries onError: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2018-10-18 01:10:54,987 INFO datanode.DataNode (DataXceiver.java:writeBlock(738)) - Receiving BP-1776487077-172.27.79.197-1539042949074:blk_1073745094_4442 src: /172.27.87.64:34388 dest: /172.27.52.130:50010
2018-10-18 01:10:54,990 INFO impl.FsDatasetImpl (FsDatasetImpl.java:convertTemporaryToRbw(1534)) - Convert BP-1776487077-172.27.79.197-1539042949074:blk_1073745094_4442 from Temporary to RBW, visible length=319488
2018-10-18 01:10:55,013 INFO datanode.DataNode (DataXceiver.java:writeBlock(738)) - Receiving BP-1776487077-172.27.79.197-1539042949074:blk_1073745094_4442 src: /172.27.68.128:46756 dest: /172.27.52.130:50010
2018-10-18 01:10:55,014 INFO impl.FsDatasetImpl (FsDatasetImpl.java:recoverRbw(1440)) - Recover RBW replica BP-1776487077-172.27.79.197-1539042949074:blk_1073745094_4442
2018-10-18 01:10:55,014 INFO impl.FsDatasetImpl (FsDatasetImpl.java:recoverRbw(1456)) - At ctr-e138-1518143905142-510793-01-000007.hwx.site:50010, Recovering ReplicaBeingWritten, blk_1073745094_4442, RBW
getNumBytes() = 319488
getBytesOnDisk() = 319488
getVisibleLength()= 319488
getVolume() = /grid/0/hadoop/hdfs/data
getBlockURI() = file:/grid/0/hadoop/hdfs/data/current/BP-1776487077-172.27.79.197-1539042949074/current/rbw/blk_1073745094
bytesAcked=319488
bytesOnDisk=319488
2018-10-18 01:10:55,099 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:55,288 WARN client.GrpcClientProtocolService (LogUtils.java:warn(135)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1-1: onError: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2018-10-18 01:10:56,273 WARN client.GrpcClientProtocolService (LogUtils.java:warn(135)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1-2: onError: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
2018-10-18 01:10:56,307 WARN server.GrpcServerProtocolService (LogUtils.java:warn(137)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed requestVote a698a509-0816-4719-9a0f-36d1b503c249->69d15283-4e2e-4c30-a028-f2bad0f83cc1#0
org.apache.ratis.protocol.GroupMismatchException: 69d15283-4e2e-4c30-a028-f2bad0f83cc1: group-80012DD9FE74 not found.
at org.apache.ratis.server.impl.RaftServerProxy$ImplMap.get(RaftServerProxy.java:114)
at org.apache.ratis.server.impl.RaftServerProxy.getImplFuture(RaftServerProxy.java:252)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:261)
at org.apache.ratis.server.impl.RaftServerProxy.getImpl(RaftServerProxy.java:256)
at org.apache.ratis.server.impl.RaftServerProxy.requestVote(RaftServerProxy.java:411)
at org.apache.ratis.grpc.server.GrpcServerProtocolService.requestVote(GrpcServerProtocolService.java:54)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$MethodHandlers.invoke(RaftServerProtocolServiceGrpc.java:319)
at org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283)
at org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707)
at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:56,579 INFO impl.FollowerState (FollowerState.java:run(76)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1 changes to CANDIDATE, lastRpcTime:2021, electionTimeout:1043ms
2018-10-18 01:10:56,579 INFO impl.RoleInfo (RoleInfo.java:shutdownFollowerState(117)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: shutdown FollowerState
2018-10-18 01:10:56,579 INFO impl.RaftServerImpl (RaftServerImpl.java:setRole(154)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1 changes role from FOLLOWER to CANDIDATE at term 1 for changeToCandidate
2018-10-18 01:10:56,580 INFO impl.RoleInfo (RoleInfo.java:updateAndGet(139)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: start LeaderElection
2018-10-18 01:10:56,581 INFO impl.RaftServerImpl (ServerState.java:setLeader(260)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: change Leader from 41636d6e-31cc-419b-bd8d-1a7e5e7d7edd to null at term 1 for initElection
2018-10-18 01:10:56,582 INFO impl.LeaderElection (LeaderElection.java:askForVotes(127)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: begin an election in Term 2
2018-10-18 01:10:56,590 INFO impl.LeaderElection (LeaderElection.java:waitForResults(230)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1 got exception when requesting votes: {}
java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:214)
at org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:146)
at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:102)
Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:222)
at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:203)
at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:132)
at org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265)
at org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:61)
at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:150)
at org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$0(LeaderElection.java:188)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.27.23.193:9858
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:325)
at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:634)
at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)
at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 more
Caused by: java.net.ConnectException: Connection refused
... 11 more
2018-10-18 01:10:56,594 INFO impl.LeaderElection (LeaderElection.java:logAndReturn(46)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Election PASSED; received 1 response(s) [69d15283-4e2e-4c30-a028-f2bad0f83cc1<-a698a509-0816-4719-9a0f-36d1b503c249#0:OK-t2] and 1 exception(s); 69d15283-4e2e-4c30-a028-f2bad0f83cc1:t2, leader=null, voted=69d15283-4e2e-4c30-a028-f2bad0f83cc1, raftlog=[(t:1, i:1)], conf=0: [41636d6e-31cc-419b-bd8d-1a7e5e7d7edd:172.27.23.193:9858, 69d15283-4e2e-4c30-a028-f2bad0f83cc1:172.27.52.130:9858, a698a509-0816-4719-9a0f-36d1b503c249:172.27.17.133:9858], old=null
2018-10-18 01:10:56,594 INFO impl.LeaderElection (LeaderElection.java:logAndReturn(52)) - 0: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2018-10-18 01:10:56,596 INFO impl.RoleInfo (RoleInfo.java:shutdownLeaderElection(130)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: shutdown LeaderElection
2018-10-18 01:10:56,596 INFO impl.RaftServerImpl (RaftServerImpl.java:setRole(154)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1 changes role from CANDIDATE to LEADER at term 2 for changeToLeader
2018-10-18 01:10:56,596 INFO impl.RaftServerImpl (ServerState.java:setLeader(260)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: change Leader from null to 69d15283-4e2e-4c30-a028-f2bad0f83cc1 at term 2 for becomeLeader, leader elected after 15ms
2018-10-18 01:10:56,600 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.staging.catchup.gap = 1000 (default)
2018-10-18 01:10:56,600 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.rpc.sleep.time = 25 ms (default)
2018-10-18 01:10:56,617 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.log.appender.buffer.capacity = 134217728 (custom)
2018-10-18 01:10:56,618 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.log.appender.batch.enabled = true (custom)
2018-10-18 01:10:56,618 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.log.appender.snapshot.chunk.size.max = 16MB (=16777216) (default)
2018-10-18 01:10:56,621 INFO grpc.GrpcConfigKeys$Server (ConfUtils.java:logGet(43)) - raft.grpc.server.leader.outstanding.appends.max = 128 (default)
2018-10-18 01:10:56,623 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.rpc.request.timeout = 3000 ms (custom)
2018-10-18 01:10:56,623 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.log.appender.buffer.capacity = 134217728 (custom)
2018-10-18 01:10:56,623 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.log.appender.batch.enabled = true (custom)
2018-10-18 01:10:56,624 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.log.appender.snapshot.chunk.size.max = 16MB (=16777216) (default)
2018-10-18 01:10:56,624 INFO grpc.GrpcConfigKeys$Server (ConfUtils.java:logGet(43)) - raft.grpc.server.leader.outstanding.appends.max = 128 (default)
2018-10-18 01:10:56,624 INFO server.RaftServerConfigKeys (ConfUtils.java:logGet(43)) - raft.server.rpc.request.timeout = 3000 ms (custom)
2018-10-18 01:10:56,625 INFO impl.RoleInfo (RoleInfo.java:updateAndGet(139)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: start LeaderState
2018-10-18 01:10:56,629 INFO storage.RaftLogWorker (RaftLogWorker.java:rollLogSegment(258)) - Rolling segment:69d15283-4e2e-4c30-a028-f2bad0f83cc1-RaftLogWorker index to:2
2018-10-18 01:10:56,630 INFO impl.RaftServerImpl (ServerState.java:setRaftConf(350)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: set configuration 2: [41636d6e-31cc-419b-bd8d-1a7e5e7d7edd:172.27.23.193:9858, 69d15283-4e2e-4c30-a028-f2bad0f83cc1:172.27.52.130:9858, a698a509-0816-4719-9a0f-36d1b503c249:172.27.17.133:9858], old=null at 2
2018-10-18 01:10:56,645 WARN server.GrpcLogAppender (LogUtils.java:warn(135)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: Failed appendEntries to 41636d6e-31cc-419b-bd8d-1a7e5e7d7edd:172.27.23.193:9858: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
2018-10-18 01:10:57,100 ERROR impl.RaftServerImpl (RaftServerImpl.java:applyLogToStateMachine(1096)) - 69d15283-4e2e-4c30-a028-f2bad0f83cc1: applyTransaction failed for index:1 proto:(t:1, i:1)SMLOGENTRY, client-137A301511FA, cid=0
2018-10-18 01:10:57,102 ERROR impl.StateMachineUpdater (ExitUtils.java:terminate(86)) - Terminating with exit status 2: StateMachineUpdater-69d15283-4e2e-4c30-a028-f2bad0f83cc1: the StateMachineUpdater hits Throwable
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.CompletableFuture$AsyncSupply@7ee4907b rejected from java.util.concurrent.ThreadPoolExecutor@702371d7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668)
at java.util.concurrent.CompletableFuture.asyncSupplyStage(CompletableFuture.java:1604)
at java.util.concurrent.CompletableFuture.supplyAsync(CompletableFuture.java:1830)
at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.applyTransaction(ContainerStateMachine.java:433)
at org.apache.ratis.server.impl.RaftServerImpl.applyLogToStateMachine(RaftServerImpl.java:1093)
at org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:148)
at java.lang.Thread.run(Thread.java:748)
2018-10-18 01:10:57,107 WARN fs.CachingGetSpaceUsed (CachingGetSpaceUsed.java:run(183)) - Thread Interrupted waiting to refresh disk information: sleep interrupted
2018-10-18 01:10:57,108 INFO datanode.DataNode (LogAdapter.java:info(51)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at ctr-e138-1518143905142-510793-01-000007.hwx.site/172.27.52.130
************************************************************/
2018-10-18 01:10:57,129 INFO conf.Configuration (Configuration.java:removeUndeclaredTags(3298)) - Removed undeclared tags:
{code}
> Datanode shuts down on restart
> ------------------------------
>
> Key: HDDS-689
> URL: https://issues.apache.org/jira/browse/HDDS-689
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Reporter: Namit Maheshwari
> Priority: Major
>
> Restart all the 8 DNs in the cluster. 2 of them shuts down as below
> {code:java}
> 2018-10-18 01:10:57,102 ERROR impl.StateMachineUpdater (ExitUtils.java:terminate(86)) - Terminating with exit status 2: StateMachineUpdater-69d15283-4e2e-4c30-a028-f2bad0f83cc1: the StateMachineUpdater hits Throwable
> java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.CompletableFuture$AsyncSupply@7ee4907b rejected from java.util.concurrent.ThreadPoolExecutor@702371d7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
> at java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668)
> at java.util.concurrent.CompletableFuture.asyncSupplyStage(CompletableFuture.java:1604)
> at java.util.concurrent.CompletableFuture.supplyAsync(CompletableFuture.java:1830)
> at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.applyTransaction(ContainerStateMachine.java:433)
> at org.apache.ratis.server.impl.RaftServerImpl.applyLogToStateMachine(RaftServerImpl.java:1093)
> at org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:148)
> at java.lang.Thread.run(Thread.java:748)
> 2018-10-18 01:10:57,107 WARN fs.CachingGetSpaceUsed (CachingGetSpaceUsed.java:run(183)) - Thread Interrupted waiting to refresh disk information: sleep interrupted
> 2018-10-18 01:10:57,108 INFO datanode.DataNode (LogAdapter.java:info(51)) - SHUTDOWN_MSG:
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org