You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2019/03/22 10:40:00 UTC
[jira] [Work logged] (HDDS-1309) remove extra logging to stdout
while trying to read key when one of the datanodes is down
[ https://issues.apache.org/jira/browse/HDDS-1309?focusedWorklogId=217183&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-217183 ]
ASF GitHub Bot logged work on HDDS-1309:
----------------------------------------
Author: ASF GitHub Bot
Created on: 22/Mar/19 10:39
Start Date: 22/Mar/19 10:39
Worklog Time Spent: 10m
Work Description: nilotpalnandi commented on pull request #638: HDDS-1309 . change logging from warn to debug in XceiverClient. Contr…
URL: https://github.com/apache/hadoop/pull/638
…ibuted by Nilotpal Nandi.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 217183)
Time Spent: 10m
Remaining Estimate: 0h
> remove extra logging to stdout while trying to read key when one of the datanodes is down
> -----------------------------------------------------------------------------------------
>
> Key: HDDS-1309
> URL: https://issues.apache.org/jira/browse/HDDS-1309
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Reporter: Nilotpal Nandi
> Assignee: Nilotpal Nandi
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> steps taken :
> ---------------------
> # wrote of key in 3 datanode cluster.
> # shutdown a datanode
> # read the key
> observation:
> ---------------------
> read was successful.
> But , it should not print all logs in client's STDOUT.
>
> client's output:
> ---------------------
> {noformat}
> 2019-03-19 09:49:14,329|INFO|MainThread|ozone.py:92 - getKey()|Running get key WriteOzoneFile_1552988856 in test-volume1-1552988834/test-bucket1-1552988834
> 2019-03-19 09:49:14,331|INFO|MainThread|machine.py:165 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|RUNNING: ssh -l root -i /tmp/hw-qe-keypair.pem -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ctr-e139-1542663976389-88627-01-000002.hwx.site "sudo su - -c \"/usr/hdp/current/hadoop-ozone/bin/ozone sh key get /test-volume1-1552988834/test-bucket1-1552988834/WriteOzoneFile_1552988856 /tmp/OzonePipelineTest/WriteOzoneFileGet_1552988856\" root"
> 2019-03-19 09:49:17,915|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|SLF4J: Class path contains multiple SLF4J bindings.
> 2019-03-19 09:49:17,916|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|SLF4J: Found binding in [jar:file:/usr/hdp/3.0.100.0-348/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 2019-03-19 09:49:17,916|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|SLF4J: Found binding in [jar:file:/usr/hdp/3.0.100.0-348/hadoop-ozone/share/ozone/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 2019-03-19 09:49:17,917|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> 2019-03-19 09:49:17,993|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 2019-03-19 09:49:21,371|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|19/03/19 09:49:21 WARN scm.XceiverClientGrpc: Failed to execute command cmdType: GetBlock
> 2019-03-19 09:49:21,372|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|traceID: "f5bb707b-b0ef-4c21-a6ea-910b72221cfd"
> 2019-03-19 09:49:21,372|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 1
> 2019-03-19 09:49:21,372|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|datanodeUuid: "1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c"
> 2019-03-19 09:49:21,372|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|getBlock {
> 2019-03-19 09:49:21,372|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockID {
> 2019-03-19 09:49:21,373|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 1
> 2019-03-19 09:49:21,373|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|localID: 101776679626145792
> 2019-03-19 09:49:21,373|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockCommitSequenceId: 23
> 2019-03-19 09:49:21,373|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:21,374|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:21,374|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|on datanode 1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c
> 2019-03-19 09:49:21,374|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:21,375|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2019-03-19 09:49:21,375|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> 2019-03-19 09:49:21,375|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:247)
> 2019-03-19 09:49:21,375|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:204)
> 2019-03-19 09:49:21,376|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:118)
> 2019-03-19 09:49:21,376|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream.getFromOmKeyInfo(KeyInputStream.java:305)
> 2019-03-19 09:49:21,376|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.rpc.RpcClient.getKey(RpcClient.java:608)
> 2019-03-19 09:49:21,376|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.OzoneBucket.readKey(OzoneBucket.java:284)
> 2019-03-19 09:49:21,377|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:95)
> 2019-03-19 09:49:21,377|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:48)
> 2019-03-19 09:49:21,377|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.execute(CommandLine.java:919)
> 2019-03-19 09:49:21,378|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.access$700(CommandLine.java:104)
> 2019-03-19 09:49:21,378|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
> 2019-03-19 09:49:21,378|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
> 2019-03-19 09:49:21,378|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
> 2019-03-19 09:49:21,379|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
> 2019-03-19 09:49:21,379|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
> 2019-03-19 09:49:21,379|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
> 2019-03-19 09:49:21,380|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.execute(Shell.java:84)
> 2019-03-19 09:49:21,380|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
> 2019-03-19 09:49:21,380|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:95)
> 2019-03-19 09:49:21,381|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:21,381|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:526)
> 2019-03-19 09:49:21,382|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434)
> 2019-03-19 09:49:21,382|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:21,382|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:21,383|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:21,383|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678)
> 2019-03-19 09:49:21,384|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:21,384|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:21,384|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:21,385|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
> 2019-03-19 09:49:21,385|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> 2019-03-19 09:49:21,385|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> 2019-03-19 09:49:21,386|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> 2019-03-19 09:49:21,386|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> 2019-03-19 09:49:21,386|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> 2019-03-19 09:49:21,387|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2019-03-19 09:49:21,387|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> 2019-03-19 09:49:21,387|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2019-03-19 09:49:21,387|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2019-03-19 09:49:21,388|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.lang.Thread.run(Thread.java:748)
> 2019-03-19 09:49:21,388|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.27.89.128:9859
> 2019-03-19 09:49:21,388|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 2019-03-19 09:49:21,388|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> 2019-03-19 09:49:21,389|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
> 2019-03-19 09:49:21,389|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
> 2019-03-19 09:49:21,389|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632)
> 2019-03-19 09:49:21,390|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
> 2019-03-19 09:49:21,390|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
> 2019-03-19 09:49:21,390|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> 2019-03-19 09:49:21,391|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> 2019-03-19 09:49:21,391|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2019-03-19 09:49:21,391|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 1 more
> 2019-03-19 09:49:21,392|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: java.net.ConnectException: Connection refused
> 2019-03-19 09:49:21,392|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 11 more
> 2019-03-19 09:49:21,564|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|19/03/19 09:49:21 WARN scm.XceiverClientGrpc: Failed to execute command cmdType: GetBlock
> 2019-03-19 09:49:21,564|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|traceID: "f5bb707b-b0ef-4c21-a6ea-910b72221cfd"
> 2019-03-19 09:49:21,565|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 2
> 2019-03-19 09:49:21,565|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|datanodeUuid: "1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c"
> 2019-03-19 09:49:21,566|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|getBlock {
> 2019-03-19 09:49:21,566|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockID {
> 2019-03-19 09:49:21,566|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 2
> 2019-03-19 09:49:21,566|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|localID: 101776681938518017
> 2019-03-19 09:49:21,567|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockCommitSequenceId: 43
> 2019-03-19 09:49:21,567|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:21,568|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:21,568|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|on datanode 1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c
> 2019-03-19 09:49:21,568|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:21,569|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2019-03-19 09:49:21,569|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> 2019-03-19 09:49:21,569|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:247)
> 2019-03-19 09:49:21,570|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:204)
> 2019-03-19 09:49:21,570|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:118)
> 2019-03-19 09:49:21,570|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream.getFromOmKeyInfo(KeyInputStream.java:305)
> 2019-03-19 09:49:21,571|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.rpc.RpcClient.getKey(RpcClient.java:608)
> 2019-03-19 09:49:21,571|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.OzoneBucket.readKey(OzoneBucket.java:284)
> 2019-03-19 09:49:21,571|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:95)
> 2019-03-19 09:49:21,571|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:48)
> 2019-03-19 09:49:21,572|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.execute(CommandLine.java:919)
> 2019-03-19 09:49:21,572|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.access$700(CommandLine.java:104)
> 2019-03-19 09:49:21,572|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
> 2019-03-19 09:49:21,572|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
> 2019-03-19 09:49:21,573|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
> 2019-03-19 09:49:21,573|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
> 2019-03-19 09:49:21,573|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
> 2019-03-19 09:49:21,574|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
> 2019-03-19 09:49:21,574|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.execute(Shell.java:84)
> 2019-03-19 09:49:21,574|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
> 2019-03-19 09:49:21,574|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:95)
> 2019-03-19 09:49:21,575|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:21,575|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:526)
> 2019-03-19 09:49:21,575|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434)
> 2019-03-19 09:49:21,575|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:21,575|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:21,576|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:21,576|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678)
> 2019-03-19 09:49:21,576|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:21,576|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:21,577|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:21,577|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
> 2019-03-19 09:49:21,577|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> 2019-03-19 09:49:21,577|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> 2019-03-19 09:49:21,578|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> 2019-03-19 09:49:21,578|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> 2019-03-19 09:49:21,578|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> 2019-03-19 09:49:21,578|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2019-03-19 09:49:21,579|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> 2019-03-19 09:49:21,580|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2019-03-19 09:49:21,580|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2019-03-19 09:49:21,580|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.lang.Thread.run(Thread.java:748)
> 2019-03-19 09:49:21,580|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.27.89.128:9859
> 2019-03-19 09:49:21,581|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 2019-03-19 09:49:21,581|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> 2019-03-19 09:49:21,581|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
> 2019-03-19 09:49:21,582|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
> 2019-03-19 09:49:21,582|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632)
> 2019-03-19 09:49:21,583|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
> 2019-03-19 09:49:21,583|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
> 2019-03-19 09:49:21,583|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> 2019-03-19 09:49:21,584|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> 2019-03-19 09:49:21,584|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2019-03-19 09:49:21,584|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 1 more
> 2019-03-19 09:49:21,585|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: java.net.ConnectException: Connection refused
> 2019-03-19 09:49:21,585|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 11 more
> 2019-03-19 09:49:21,807|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|19/03/19 09:49:21 WARN scm.XceiverClientGrpc: Failed to execute command cmdType: ReadChunk
> 2019-03-19 09:49:21,807|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|traceID: "f5bb707b-b0ef-4c21-a6ea-910b72221cfd"
> 2019-03-19 09:49:21,807|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 1
> 2019-03-19 09:49:21,808|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|datanodeUuid: "1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c"
> 2019-03-19 09:49:21,808|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|readChunk {
> 2019-03-19 09:49:21,808|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockID {
> 2019-03-19 09:49:21,809|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 1
> 2019-03-19 09:49:21,809|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|localID: 101776679626145792
> 2019-03-19 09:49:21,809|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockCommitSequenceId: 23
> 2019-03-19 09:49:21,810|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:21,810|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkData {
> 2019-03-19 09:49:21,810|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkName: "f42c4a33a0b766f6a39981411a77be83_stream_dfa8c5a9-80e8-4c2a-a371-058de7f11f64_chunk_1"
> 2019-03-19 09:49:21,811|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|offset: 0
> 2019-03-19 09:49:21,811|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|len: 16777216
> 2019-03-19 09:49:21,811|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksumData {
> 2019-03-19 09:49:21,811|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|type: SHA256
> 2019-03-19 09:49:21,811|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|bytesPerChecksum: 1048576
> 2019-03-19 09:49:21,812|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\261]\364\310\n\\\275\300\335\335\0008\314\236\332\241\273\360Qhbr\247@g\277\263i\275\224]\273"
> 2019-03-19 09:49:21,812|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "b!\034e\2616/\324&=\234\022\324m\327\305\304\313\304:\227P\035br~x<M\217tk"
> 2019-03-19 09:49:21,812|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:21,812|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "ttm `m\306Ed%\r\326\035c\362\372YF\275\032\302s\316d\024\177\034n\237\346\t"
> 2019-03-19 09:49:21,813|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:21,813|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:21,813|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:21,813|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\330\316\3634J\265\t\275\362\251%HG\245I\'\357\277\322Lr?5MI\vL\177\032\341\201\023"
> 2019-03-19 09:49:21,814|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:21,814|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:21,814|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:21,814|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "]\"\234\027\340\000Y<\257\004\271+1\233S2\274\226Ww \325>\030X\240\344\314\004\314\344\262"
> 2019-03-19 09:49:21,815|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "%\302\222\025\217\234\237\306\367\003B\001\324,\220p\037\3225\237\215\023a\222\272:\272\346\361\3044\370"
> 2019-03-19 09:49:21,815|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "-\f\v\020\223\030S\321M\334.I\277<\370r\353\362O\371\354v\311\v#\306\200+\245\336{\312"
> 2019-03-19 09:49:21,815|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\336u\335\277a\363\216\025\375\034\242\004F\205.&(f\246Nj\027\354\v\203ZF\245&b\356\f"
> 2019-03-19 09:49:21,816|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\242T\356\306b\220X\232\356\3318<\035\005s9\004V:\370S\fg=\221\035\217\253\342\235\266m"
> 2019-03-19 09:49:21,816|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:21,816|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:21,817|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:21,817|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|on datanode 1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c
> 2019-03-19 09:49:21,817|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:21,818|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2019-03-19 09:49:21,818|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> 2019-03-19 09:49:21,818|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:247)
> 2019-03-19 09:49:21,819|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:217)
> 2019-03-19 09:49:21,819|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:256)
> 2019-03-19 09:49:21,819|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:292)
> 2019-03-19 09:49:21,820|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.prepareRead(BlockInputStream.java:226)
> 2019-03-19 09:49:21,820|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:150)
> 2019-03-19 09:49:21,820|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.read(KeyInputStream.java:232)
> 2019-03-19 09:49:21,820|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:126)
> 2019-03-19 09:49:21,821|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47)
> 2019-03-19 09:49:21,821|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.io.InputStream.read(InputStream.java:101)
> 2019-03-19 09:49:21,821|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:94)
> 2019-03-19 09:49:21,822|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:98)
> 2019-03-19 09:49:21,822|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:48)
> 2019-03-19 09:49:21,822|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.execute(CommandLine.java:919)
> 2019-03-19 09:49:21,823|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.access$700(CommandLine.java:104)
> 2019-03-19 09:49:21,823|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
> 2019-03-19 09:49:21,823|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
> 2019-03-19 09:49:21,824|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
> 2019-03-19 09:49:21,824|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
> 2019-03-19 09:49:21,824|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
> 2019-03-19 09:49:21,825|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
> 2019-03-19 09:49:21,825|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.execute(Shell.java:84)
> 2019-03-19 09:49:21,825|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
> 2019-03-19 09:49:21,826|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:95)
> 2019-03-19 09:49:21,826|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:21,826|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:526)
> 2019-03-19 09:49:21,827|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434)
> 2019-03-19 09:49:21,827|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:21,828|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:21,828|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:21,828|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678)
> 2019-03-19 09:49:21,829|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:21,829|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:21,829|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:21,830|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
> 2019-03-19 09:49:21,830|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> 2019-03-19 09:49:21,831|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> 2019-03-19 09:49:21,831|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> 2019-03-19 09:49:21,831|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> 2019-03-19 09:49:21,832|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> 2019-03-19 09:49:21,832|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2019-03-19 09:49:21,832|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> 2019-03-19 09:49:21,832|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2019-03-19 09:49:21,832|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2019-03-19 09:49:21,832|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.lang.Thread.run(Thread.java:748)
> 2019-03-19 09:49:21,833|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.27.89.128:9859
> 2019-03-19 09:49:21,833|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 2019-03-19 09:49:21,833|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> 2019-03-19 09:49:21,833|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
> 2019-03-19 09:49:21,834|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
> 2019-03-19 09:49:21,834|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632)
> 2019-03-19 09:49:21,834|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
> 2019-03-19 09:49:21,834|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
> 2019-03-19 09:49:21,834|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> 2019-03-19 09:49:21,835|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> 2019-03-19 09:49:21,835|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2019-03-19 09:49:21,835|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 1 more
> 2019-03-19 09:49:21,835|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: java.net.ConnectException: Connection refused
> 2019-03-19 09:49:21,835|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 11 more
> 2019-03-19 09:49:22,115|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|19/03/19 09:49:21 WARN scm.XceiverClientGrpc: Failed to execute command cmdType: ReadChunk
> 2019-03-19 09:49:22,115|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|traceID: "f5bb707b-b0ef-4c21-a6ea-910b72221cfd"
> 2019-03-19 09:49:22,115|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 1
> 2019-03-19 09:49:22,116|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|datanodeUuid: "1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c"
> 2019-03-19 09:49:22,116|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|readChunk {
> 2019-03-19 09:49:22,116|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockID {
> 2019-03-19 09:49:22,116|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 1
> 2019-03-19 09:49:22,117|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|localID: 101776679626145792
> 2019-03-19 09:49:22,117|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockCommitSequenceId: 23
> 2019-03-19 09:49:22,117|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:22,117|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkData {
> 2019-03-19 09:49:22,117|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkName: "f42c4a33a0b766f6a39981411a77be83_stream_dfa8c5a9-80e8-4c2a-a371-058de7f11f64_chunk_2"
> 2019-03-19 09:49:22,118|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|offset: 0
> 2019-03-19 09:49:22,118|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|len: 16777216
> 2019-03-19 09:49:22,118|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksumData {
> 2019-03-19 09:49:22,118|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|type: SHA256
> 2019-03-19 09:49:22,119|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|bytesPerChecksum: 1048576
> 2019-03-19 09:49:22,119|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\327^\207\361~\207\245\321\273\327\a\300.\337\344\"^{U\000\261\334\256|`\223\227\341I\233~\201"
> 2019-03-19 09:49:22,119|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "e|g\357\222]\\\353\203\375\230\032\351\266\274]\331\246\362X1!\035\214\2766\222\2321\235\370f"
> 2019-03-19 09:49:22,120|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\333()a8!l\271\325\273d\v\272\214>!\315\354\245|3S(\001\235\324\221\356\216\372\356\324"
> 2019-03-19 09:49:22,120|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\254\265\254)\232h\224\320\n\261D\352\033[\245\263\3011\211/|5@\367\23485p\227\315jX"
> 2019-03-19 09:49:22,120|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\021\aye_X\206R+t(\037\345\272e\314\314\275\261k\360,ff\376n\210R\356\027\237\026"
> 2019-03-19 09:49:22,121|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\341\237\237\t\251\217\"v\202\231~F\244\355[\vK\b\221\235\n\277\373\343\331\375S\365\237+J\357"
> 2019-03-19 09:49:22,121|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\304\363l\017\006\315\030\023\246\274\tO\0035+J\353#\034v\350\3345\bvo\367J\361\361\201\302"
> 2019-03-19 09:49:22,122|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "`\275\034r\3157I7\272\017r\343\0230>\250R&\253\371\347\266k\245d\b(\335=\v\021\177"
> 2019-03-19 09:49:22,122|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\375\213\224$\301\004$\230\312Vd\'\265\336OT\211e\v\"\303\'\303\032\034\\\201I\032\313\363\352"
> 2019-03-19 09:49:22,122|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "[\344\271\376\347\200\207\271\204\253\371t\2401\324\304\245\277\236\332\326\362\365\327\374\226\2046t<kT"
> 2019-03-19 09:49:22,123|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\344MJ\354b[\202\002u~\n\220\2703s\004~\211B?\036\373\0178;\247\250\311\214\310\241\361"
> 2019-03-19 09:49:22,123|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\205\247\220\216On\262\227\324\261`R@\v\221X\236\021\243\316{\232\320\032\212\207\344\355\377M\325E"
> 2019-03-19 09:49:22,124|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\361\2017\217\347v\233|\267\346\325\236\270s\214\326\204I\222\334\212^\266t\325\361y\211\213\303F\342"
> 2019-03-19 09:49:22,124|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\322\313q\365\252\025\376\201q\356\216\311\226\rrX\311>\v\361\021m\337\254v{\201\244]i\225\225"
> 2019-03-19 09:49:22,124|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\222\332C\270\371\'6\017D\037y\200\321\354|\230TP\\\243\352\274\003\fr\031\223\371!H\025\""
> 2019-03-19 09:49:22,125|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\252\231D&\025\005\335\037o=2\205\026\277\234u\033\251\350\373j\321\\}\222\351f\306\035YH\246"
> 2019-03-19 09:49:22,125|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:22,125|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:22,126|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:22,126|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|on datanode 1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c
> 2019-03-19 09:49:22,126|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:22,127|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2019-03-19 09:49:22,127|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> 2019-03-19 09:49:22,127|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:247)
> 2019-03-19 09:49:22,128|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:217)
> 2019-03-19 09:49:22,128|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:256)
> 2019-03-19 09:49:22,128|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:292)
> 2019-03-19 09:49:22,129|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.prepareRead(BlockInputStream.java:226)
> 2019-03-19 09:49:22,129|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:150)
> 2019-03-19 09:49:22,129|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.read(KeyInputStream.java:232)
> 2019-03-19 09:49:22,130|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:126)
> 2019-03-19 09:49:22,130|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47)
> 2019-03-19 09:49:22,131|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.io.InputStream.read(InputStream.java:101)
> 2019-03-19 09:49:22,131|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
> 2019-03-19 09:49:22,131|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:98)
> 2019-03-19 09:49:22,132|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:48)
> 2019-03-19 09:49:22,132|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.execute(CommandLine.java:919)
> 2019-03-19 09:49:22,132|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.access$700(CommandLine.java:104)
> 2019-03-19 09:49:22,133|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
> 2019-03-19 09:49:22,133|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
> 2019-03-19 09:49:22,133|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
> 2019-03-19 09:49:22,134|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
> 2019-03-19 09:49:22,134|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
> 2019-03-19 09:49:22,135|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
> 2019-03-19 09:49:22,135|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.execute(Shell.java:84)
> 2019-03-19 09:49:22,135|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
> 2019-03-19 09:49:22,136|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:95)
> 2019-03-19 09:49:22,136|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:22,136|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:526)
> 2019-03-19 09:49:22,137|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434)
> 2019-03-19 09:49:22,137|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:22,138|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:22,138|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:22,138|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678)
> 2019-03-19 09:49:22,139|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:22,139|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:22,139|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:22,140|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
> 2019-03-19 09:49:22,140|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> 2019-03-19 09:49:22,140|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> 2019-03-19 09:49:22,140|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> 2019-03-19 09:49:22,141|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> 2019-03-19 09:49:22,141|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> 2019-03-19 09:49:22,142|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2019-03-19 09:49:22,142|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> 2019-03-19 09:49:22,142|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2019-03-19 09:49:22,143|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2019-03-19 09:49:22,143|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.lang.Thread.run(Thread.java:748)
> 2019-03-19 09:49:22,143|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.27.89.128:9859
> 2019-03-19 09:49:22,144|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 2019-03-19 09:49:22,144|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> 2019-03-19 09:49:22,144|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
> 2019-03-19 09:49:22,145|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
> 2019-03-19 09:49:22,145|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632)
> 2019-03-19 09:49:22,145|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
> 2019-03-19 09:49:22,146|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
> 2019-03-19 09:49:22,146|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> 2019-03-19 09:49:22,146|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> 2019-03-19 09:49:22,147|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2019-03-19 09:49:22,147|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 1 more
> 2019-03-19 09:49:22,147|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: java.net.ConnectException: Connection refused
> 2019-03-19 09:49:22,148|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 11 more
> 2019-03-19 09:49:22,424|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|19/03/19 09:49:22 WARN scm.XceiverClientGrpc: Failed to execute command cmdType: ReadChunk
> 2019-03-19 09:49:22,425|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|traceID: "f5bb707b-b0ef-4c21-a6ea-910b72221cfd"
> 2019-03-19 09:49:22,425|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 1
> 2019-03-19 09:49:22,425|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|datanodeUuid: "1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c"
> 2019-03-19 09:49:22,426|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|readChunk {
> 2019-03-19 09:49:22,426|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockID {
> 2019-03-19 09:49:22,426|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 1
> 2019-03-19 09:49:22,427|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|localID: 101776679626145792
> 2019-03-19 09:49:22,427|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockCommitSequenceId: 23
> 2019-03-19 09:49:22,428|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:22,428|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkData {
> 2019-03-19 09:49:22,428|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkName: "f42c4a33a0b766f6a39981411a77be83_stream_dfa8c5a9-80e8-4c2a-a371-058de7f11f64_chunk_3"
> 2019-03-19 09:49:22,428|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|offset: 0
> 2019-03-19 09:49:22,429|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|len: 16777216
> 2019-03-19 09:49:22,429|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksumData {
> 2019-03-19 09:49:22,429|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|type: SHA256
> 2019-03-19 09:49:22,430|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|bytesPerChecksum: 1048576
> 2019-03-19 09:49:22,430|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\vg\214\271\335\357v\312A\'\311\025\002\313G\3471\204\233\274\1778\307\222I.\340\1771Nxu"
> 2019-03-19 09:49:22,430|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "U\006e9\246\2118lm{\024\360j\033/\206q\372\235\376\246\224$\273m\252\240\246\033\273\201\003"
> 2019-03-19 09:49:22,431|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\205?=\274~\336\356\\\252\nL\220\252\352\016\266\370\025\033\352\205L\330\320\333\032\371\264\272\371\330\223"
> 2019-03-19 09:49:22,431|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\357x\217N\224\232\323\206k\037\227\262L\224\211\360\004t\235\225\2155q*\342q\024\360Lf\250\311"
> 2019-03-19 09:49:22,432|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\257_Z\022g\214O\343Y9,v\372\256\233\037\212\030\314:\315\247[\300V6]%\203\377%n"
> 2019-03-19 09:49:22,432|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "I;y\002p\307U\037F \034\027\230,\1775\311(\344\250\246\343\321\363%srR\301=\303."
> 2019-03-19 09:49:22,432|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "<5\357R\027\312G\005\252s\332\230\332\001YO\206\350\266\262\244\222\371\3706\212\021\373\351\244\235/"
> 2019-03-19 09:49:22,432|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\2602\343_\3417\267=/\351\310\3404\373\371L\327\272\264\334\004!\323\a%E\336\324\376\212\256G"
> 2019-03-19 09:49:22,433|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "i\332\212\355\346\364O\256\236PT\316\375\224\250\034\314\224\001\357\243\316o\376\223\241\261\036\326\2025-"
> 2019-03-19 09:49:22,433|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\340\247\277\355\350\214\027\321\036\301\246\362&^\233\262\344&tP\217\234\301:\364\202\233[\237\020\255\377"
> 2019-03-19 09:49:22,433|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\276XE\344\240Hp\0042\'\263\300[x<\355\252\t\250\332\255\017F\260\2041L23\222\302C"
> 2019-03-19 09:49:22,434|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "Z\205\331\371\215\t\201~`\352z\315\351I\251\235\341\325\272\364X\f\353\310B\371\341{~)\377\305"
> 2019-03-19 09:49:22,434|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:22,435|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "{\235\035o\230Fy\355\276\246\273\210q;\2372\2049\347\337rdK\215/\326#~\356\336\035\317"
> 2019-03-19 09:49:22,435|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\000\245\336\307!\371R\202\343q\335\240\231\247i\343\027\240\341\333U\211D\315\305\235\253\026z\345\360\254"
> 2019-03-19 09:49:22,435|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\235\217\223\355\275\341_`\017\340\333\006\251S\372\352\336=\261\317V\331\200\304\366\324\264|m\316%\023"
> 2019-03-19 09:49:22,436|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:22,436|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:22,436|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:22,437|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|on datanode 1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c
> 2019-03-19 09:49:22,437|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:22,437|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2019-03-19 09:49:22,438|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> 2019-03-19 09:49:22,438|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:247)
> 2019-03-19 09:49:22,439|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:217)
> 2019-03-19 09:49:22,439|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:256)
> 2019-03-19 09:49:22,439|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:292)
> 2019-03-19 09:49:22,439|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.prepareRead(BlockInputStream.java:226)
> 2019-03-19 09:49:22,440|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:150)
> 2019-03-19 09:49:22,440|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.read(KeyInputStream.java:232)
> 2019-03-19 09:49:22,440|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:126)
> 2019-03-19 09:49:22,441|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47)
> 2019-03-19 09:49:22,441|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.io.InputStream.read(InputStream.java:101)
> 2019-03-19 09:49:22,441|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
> 2019-03-19 09:49:22,442|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:98)
> 2019-03-19 09:49:22,442|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:48)
> 2019-03-19 09:49:22,442|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.execute(CommandLine.java:919)
> 2019-03-19 09:49:22,443|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.access$700(CommandLine.java:104)
> 2019-03-19 09:49:22,443|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
> 2019-03-19 09:49:22,443|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
> 2019-03-19 09:49:22,444|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
> 2019-03-19 09:49:22,444|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
> 2019-03-19 09:49:22,444|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
> 2019-03-19 09:49:22,445|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
> 2019-03-19 09:49:22,445|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.execute(Shell.java:84)
> 2019-03-19 09:49:22,445|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
> 2019-03-19 09:49:22,445|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:95)
> 2019-03-19 09:49:22,446|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:22,446|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:526)
> 2019-03-19 09:49:22,447|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434)
> 2019-03-19 09:49:22,447|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:22,447|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:22,448|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:22,448|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678)
> 2019-03-19 09:49:22,449|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:22,449|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:22,449|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:22,450|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
> 2019-03-19 09:49:22,450|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> 2019-03-19 09:49:22,450|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> 2019-03-19 09:49:22,451|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> 2019-03-19 09:49:22,451|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> 2019-03-19 09:49:22,452|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> 2019-03-19 09:49:22,452|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2019-03-19 09:49:22,452|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> 2019-03-19 09:49:22,452|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2019-03-19 09:49:22,453|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2019-03-19 09:49:22,453|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.lang.Thread.run(Thread.java:748)
> 2019-03-19 09:49:22,453|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.27.89.128:9859
> 2019-03-19 09:49:22,454|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 2019-03-19 09:49:22,454|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> 2019-03-19 09:49:22,454|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
> 2019-03-19 09:49:22,455|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
> 2019-03-19 09:49:22,455|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632)
> 2019-03-19 09:49:22,455|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
> 2019-03-19 09:49:22,456|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
> 2019-03-19 09:49:22,456|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> 2019-03-19 09:49:22,456|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> 2019-03-19 09:49:22,457|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2019-03-19 09:49:22,457|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 1 more
> 2019-03-19 09:49:22,457|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: java.net.ConnectException: Connection refused
> 2019-03-19 09:49:22,457|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 11 more
> 2019-03-19 09:49:22,670|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|19/03/19 09:49:22 WARN scm.XceiverClientGrpc: Failed to execute command cmdType: ReadChunk
> 2019-03-19 09:49:22,670|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|traceID: "f5bb707b-b0ef-4c21-a6ea-910b72221cfd"
> 2019-03-19 09:49:22,670|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 1
> 2019-03-19 09:49:22,671|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|datanodeUuid: "1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c"
> 2019-03-19 09:49:22,671|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|readChunk {
> 2019-03-19 09:49:22,671|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockID {
> 2019-03-19 09:49:22,672|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 1
> 2019-03-19 09:49:22,672|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|localID: 101776679626145792
> 2019-03-19 09:49:22,672|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockCommitSequenceId: 23
> 2019-03-19 09:49:22,673|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:22,673|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkData {
> 2019-03-19 09:49:22,673|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkName: "f42c4a33a0b766f6a39981411a77be83_stream_dfa8c5a9-80e8-4c2a-a371-058de7f11f64_chunk_4"
> 2019-03-19 09:49:22,673|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|offset: 0
> 2019-03-19 09:49:22,674|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|len: 16777216
> 2019-03-19 09:49:22,674|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksumData {
> 2019-03-19 09:49:22,674|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|type: SHA256
> 2019-03-19 09:49:22,675|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|bytesPerChecksum: 1048576
> 2019-03-19 09:49:22,675|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\344\032\331\0304[,\365b\247V\252mzH\301k,\371\347;\377h\023\332*\216\252\203{r\342"
> 2019-03-19 09:49:22,676|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\261k7s)\177BHOM`\274\215y\2610\033\370\260I\372\020+\303F?=Gn\230\322\340"
> 2019-03-19 09:49:22,676|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "y\304\3100\243va\363\244YD\217c\034.\227=\365\350t\366\220\027\325-M\226qU>f\362"
> 2019-03-19 09:49:22,676|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\0279\r]\034\314\353\200`8u)\n\310\324j)B\335\335g\3266\307\246)\374\v\320\351\224\241"
> 2019-03-19 09:49:22,677|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\026\367o\266_=\350C(s\v\277vBB\241\210\342O\253\324V\302\217\241!\234\327o\315\304\343"
> 2019-03-19 09:49:22,677|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "=\331@}G\332f\334\247\216\222\357\376\223\320^\264\261iC\023\214O &\227\322\025\306(i\347"
> 2019-03-19 09:49:22,677|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\367W\230\343j\032\031\357\261\024\277\317\317g\326<\255\034\t\177\n\177U\341\353\'\026\362\020\250K~"
> 2019-03-19 09:49:22,678|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\360m\227\362\241\231\n)\242\022J\222=\324\230b\n\020\340\316^&\006\234;J/\250\037\006)\264"
> 2019-03-19 09:49:22,678|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\035\221\240\025h\t\f\025~\b\374\345\330\027\324\263\025\277\266\030\253g\2513t\264Y\277\243\247\325\242"
> 2019-03-19 09:49:22,678|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "6\370\370\037A\203\245@\207Y\203\277\303\3769-\331\\h\271\356\323\016$Y}K\307 \347N\350"
> 2019-03-19 09:49:22,679|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\241\265\206.\317p\316Q2\337I(\2550\303`\322\253i\256\310\222\272\027}\263Ky\336\242\357n"
> 2019-03-19 09:49:22,679|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\237\313C\251\353\236\352s]mo\267\364\344^u\021D\240{l\v\203g\362\2051\224\bK\364\217"
> 2019-03-19 09:49:22,679|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:22,680|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "1\247\\\374\036\217\034`]\366\374\325\022\213U\341\214^iT\205z9\312d\275\204[(\b\221\235"
> 2019-03-19 09:49:22,680|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "ti\006\\\366\031\222\343\300\353\227\251\a\033\177\301 k\304v3_\272\252t\361\273\\6\301}\n"
> 2019-03-19 09:49:22,680|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\362\202\f5\020%4\370\322\016\036\310\017C[\250\220L\3131\005{m\321\361\233\267\235:\252\231\206"
> 2019-03-19 09:49:22,681|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:22,681|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:22,681|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:22,681|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|on datanode 1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c
> 2019-03-19 09:49:22,682|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:22,682|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2019-03-19 09:49:22,682|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> 2019-03-19 09:49:22,683|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:247)
> 2019-03-19 09:49:22,683|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:217)
> 2019-03-19 09:49:22,683|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:256)
> 2019-03-19 09:49:22,684|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:292)
> 2019-03-19 09:49:22,684|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.prepareRead(BlockInputStream.java:226)
> 2019-03-19 09:49:22,684|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:150)
> 2019-03-19 09:49:22,685|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.read(KeyInputStream.java:232)
> 2019-03-19 09:49:22,685|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:126)
> 2019-03-19 09:49:22,686|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47)
> 2019-03-19 09:49:22,686|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.io.InputStream.read(InputStream.java:101)
> 2019-03-19 09:49:22,686|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
> 2019-03-19 09:49:22,687|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:98)
> 2019-03-19 09:49:22,687|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:48)
> 2019-03-19 09:49:22,687|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.execute(CommandLine.java:919)
> 2019-03-19 09:49:22,687|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.access$700(CommandLine.java:104)
> 2019-03-19 09:49:22,688|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
> 2019-03-19 09:49:22,688|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
> 2019-03-19 09:49:22,688|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
> 2019-03-19 09:49:22,689|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
> 2019-03-19 09:49:22,689|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
> 2019-03-19 09:49:22,690|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
> 2019-03-19 09:49:22,690|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.execute(Shell.java:84)
> 2019-03-19 09:49:22,691|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
> 2019-03-19 09:49:22,691|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:95)
> 2019-03-19 09:49:22,691|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:22,691|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:526)
> 2019-03-19 09:49:22,692|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434)
> 2019-03-19 09:49:22,692|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:22,693|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:22,693|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:22,694|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678)
> 2019-03-19 09:49:22,694|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:22,694|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:22,695|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:22,695|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
> 2019-03-19 09:49:22,696|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> 2019-03-19 09:49:22,696|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> 2019-03-19 09:49:22,696|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> 2019-03-19 09:49:22,697|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> 2019-03-19 09:49:22,698|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> 2019-03-19 09:49:22,698|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2019-03-19 09:49:22,698|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> 2019-03-19 09:49:22,699|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2019-03-19 09:49:22,699|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2019-03-19 09:49:22,699|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.lang.Thread.run(Thread.java:748)
> 2019-03-19 09:49:22,700|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.27.89.128:9859
> 2019-03-19 09:49:22,700|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 2019-03-19 09:49:22,701|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> 2019-03-19 09:49:22,701|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
> 2019-03-19 09:49:22,702|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
> 2019-03-19 09:49:22,702|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632)
> 2019-03-19 09:49:22,703|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
> 2019-03-19 09:49:22,703|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
> 2019-03-19 09:49:22,703|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> 2019-03-19 09:49:22,704|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> 2019-03-19 09:49:22,704|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2019-03-19 09:49:22,704|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 1 more
> 2019-03-19 09:49:22,705|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: java.net.ConnectException: Connection refused
> 2019-03-19 09:49:22,705|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 11 more
> 2019-03-19 09:49:22,852|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|19/03/19 09:49:22 WARN scm.XceiverClientGrpc: Failed to execute command cmdType: ReadChunk
> 2019-03-19 09:49:22,853|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|traceID: "f5bb707b-b0ef-4c21-a6ea-910b72221cfd"
> 2019-03-19 09:49:22,853|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 1
> 2019-03-19 09:49:22,853|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|datanodeUuid: "1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c"
> 2019-03-19 09:49:22,854|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|readChunk {
> 2019-03-19 09:49:22,854|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockID {
> 2019-03-19 09:49:22,855|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 1
> 2019-03-19 09:49:22,856|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|localID: 101776679626145792
> 2019-03-19 09:49:22,857|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockCommitSequenceId: 23
> 2019-03-19 09:49:22,857|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:22,857|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkData {
> 2019-03-19 09:49:22,858|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkName: "f42c4a33a0b766f6a39981411a77be83_stream_dfa8c5a9-80e8-4c2a-a371-058de7f11f64_chunk_5"
> 2019-03-19 09:49:22,859|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|offset: 0
> 2019-03-19 09:49:22,860|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|len: 16777216
> 2019-03-19 09:49:22,860|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksumData {
> 2019-03-19 09:49:22,860|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|type: SHA256
> 2019-03-19 09:49:22,861|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|bytesPerChecksum: 1048576
> 2019-03-19 09:49:22,862|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\316\030\312$|\243\232\324\325\022+\235B\233g\260T\244\247h$\266&\370\252\310\237\021\315k\302r"
> 2019-03-19 09:49:22,863|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\241:4\027\216,\340\200\327\3018\245\033<\2411\351R]\364\rS\003\313\355\r\244\304\223\353\245g"
> 2019-03-19 09:49:22,864|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\235n\307LM\t9\203\303\2755\354\342\320\204\260&t\277\352\"\t\241\321p\217\305\211\316\340\026\020"
> 2019-03-19 09:49:22,864|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\302EI\b,\363\355\375\031\030\b\006\037\243a\306\202e\n&?\251%+mB5\310\224\304u\307"
> 2019-03-19 09:49:22,865|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\313\243\020p\202\347R&\317\256<_\024\037>S\022\320e\ni\227,\264\250,&Qg>\353."
> 2019-03-19 09:49:22,865|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\325\206\202\365\016\377\253\002\000\206,\217ic\365\031^\321h\256\317O\324\364s\3228\334p>\310\230"
> 2019-03-19 09:49:22,865|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\332%\230\2374M\316Z\271]\215\221\203\215Q\"\243kd\267\344\242\312\307\377Pkd+x1\276"
> 2019-03-19 09:49:22,866|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\347\021)\347\033\331\232?\326\t\273\256\017\352\3526\247\241\022\244\230y|\346\235\225\336\245\376\334\363\252"
> 2019-03-19 09:49:22,866|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\025\247B\325\'\371\230 \362\263K0d\322\272\264b\241:N\211\351\022\322\r\031F\276?\267x\303"
> 2019-03-19 09:49:22,866|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "-[\356\343.C\030\221\262*\025\244\006\317f\2540\250wjb)8\347\f\357%\177QJ;N"
> 2019-03-19 09:49:22,866|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\300\321\205\304>n\200\246\251\227\2341kF\004\362\340Tz\322_\334abP;H\205\316\211\030q"
> 2019-03-19 09:49:22,867|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\261\216\245\212\031\345B\317\270\255x\001\231\376nwE\002\301s\3571T\325-V\026\313\022\217\373u"
> 2019-03-19 09:49:22,867|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:22,867|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\373\344\226[\316\f\232S\025\177{\276\370\r\202x\226Gb\346L\237!?\343\222l\236a\374k\353"
> 2019-03-19 09:49:22,867|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "v\204\265,\003\340\b\375\252C\252C\327\212\003\324\312\374L\352\223\363\005\t\320\264\362\355[st\257"
> 2019-03-19 09:49:22,868|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "F\177/.\035\024\374\325}\025;\271\v>I={\372p\273\016\311\023\225\322\366\233J\022K\370#"
> 2019-03-19 09:49:22,868|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:22,868|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:22,868|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:22,868|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|on datanode 1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c
> 2019-03-19 09:49:22,869|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:22,869|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2019-03-19 09:49:22,870|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> 2019-03-19 09:49:22,870|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:247)
> 2019-03-19 09:49:22,870|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:217)
> 2019-03-19 09:49:22,870|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:256)
> 2019-03-19 09:49:22,871|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:292)
> 2019-03-19 09:49:22,871|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.prepareRead(BlockInputStream.java:226)
> 2019-03-19 09:49:22,871|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:150)
> 2019-03-19 09:49:22,871|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.read(KeyInputStream.java:232)
> 2019-03-19 09:49:22,871|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:126)
> 2019-03-19 09:49:22,872|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47)
> 2019-03-19 09:49:22,872|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.io.InputStream.read(InputStream.java:101)
> 2019-03-19 09:49:22,872|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
> 2019-03-19 09:49:22,872|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:98)
> 2019-03-19 09:49:22,873|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:48)
> 2019-03-19 09:49:22,873|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.execute(CommandLine.java:919)
> 2019-03-19 09:49:22,873|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.access$700(CommandLine.java:104)
> 2019-03-19 09:49:22,873|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
> 2019-03-19 09:49:22,873|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
> 2019-03-19 09:49:22,874|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
> 2019-03-19 09:49:22,874|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
> 2019-03-19 09:49:22,874|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
> 2019-03-19 09:49:22,874|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
> 2019-03-19 09:49:22,874|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.execute(Shell.java:84)
> 2019-03-19 09:49:22,875|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
> 2019-03-19 09:49:22,875|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:95)
> 2019-03-19 09:49:22,875|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:22,875|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:526)
> 2019-03-19 09:49:22,876|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434)
> 2019-03-19 09:49:22,876|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:22,876|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:22,876|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:22,877|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678)
> 2019-03-19 09:49:22,877|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:22,877|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:22,878|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:22,878|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
> 2019-03-19 09:49:22,878|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> 2019-03-19 09:49:22,878|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> 2019-03-19 09:49:22,878|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> 2019-03-19 09:49:22,879|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> 2019-03-19 09:49:22,879|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> 2019-03-19 09:49:22,879|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2019-03-19 09:49:22,879|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> 2019-03-19 09:49:22,880|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2019-03-19 09:49:22,880|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2019-03-19 09:49:22,880|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.lang.Thread.run(Thread.java:748)
> 2019-03-19 09:49:22,880|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.27.89.128:9859
> 2019-03-19 09:49:22,880|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 2019-03-19 09:49:22,880|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> 2019-03-19 09:49:22,881|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
> 2019-03-19 09:49:22,881|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
> 2019-03-19 09:49:22,881|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632)
> 2019-03-19 09:49:22,882|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
> 2019-03-19 09:49:22,882|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
> 2019-03-19 09:49:22,882|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> 2019-03-19 09:49:22,882|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> 2019-03-19 09:49:22,883|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2019-03-19 09:49:22,883|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 1 more
> 2019-03-19 09:49:22,883|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: java.net.ConnectException: Connection refused
> 2019-03-19 09:49:22,883|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 11 more
> 2019-03-19 09:49:23,266|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|19/03/19 09:49:23 WARN scm.XceiverClientGrpc: Failed to execute command cmdType: ReadChunk
> 2019-03-19 09:49:23,267|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|traceID: "f5bb707b-b0ef-4c21-a6ea-910b72221cfd"
> 2019-03-19 09:49:23,267|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 1
> 2019-03-19 09:49:23,267|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|datanodeUuid: "1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c"
> 2019-03-19 09:49:23,267|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|readChunk {
> 2019-03-19 09:49:23,267|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockID {
> 2019-03-19 09:49:23,268|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 1
> 2019-03-19 09:49:23,268|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|localID: 101776679626145792
> 2019-03-19 09:49:23,268|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockCommitSequenceId: 23
> 2019-03-19 09:49:23,268|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:23,268|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkData {
> 2019-03-19 09:49:23,268|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkName: "f42c4a33a0b766f6a39981411a77be83_stream_dfa8c5a9-80e8-4c2a-a371-058de7f11f64_chunk_6"
> 2019-03-19 09:49:23,269|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|offset: 0
> 2019-03-19 09:49:23,269|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|len: 16777216
> 2019-03-19 09:49:23,269|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksumData {
> 2019-03-19 09:49:23,269|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|type: SHA256
> 2019-03-19 09:49:23,270|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|bytesPerChecksum: 1048576
> 2019-03-19 09:49:23,270|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\300,G$\272\337\177K\311\226b\307\270x0\315h\315\001\351\327\316_\016\313Y7\331U\215\333`"
> 2019-03-19 09:49:23,270|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: ">\245N\375\247\365Vk\311 \\\356\325@\336\213* \273Z5\373\312Z\352\276\003\205\313w\255\315"
> 2019-03-19 09:49:23,271|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\032\206\257\367\214\232\256\222\017o\202E\203\215\355Sp\n-\302\361\301\3734\001F\262\327\031\335\336\'"
> 2019-03-19 09:49:23,271|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\242k^?\241\2114\327d\0029\256\2030\371\241\317\205g\231\323\027v\235\370\252-\266\374\303W\264"
> 2019-03-19 09:49:23,271|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "B\243\fhv-\aRE~2\247u\263\302J.p0\f\3623\317\273\022*K\242}\205\326e"
> 2019-03-19 09:49:23,272|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\205\003\363n\236\226\177\327U\316@|\333\236\342\351\230\371\223|\034\377i\214N\025\344\252\241t\005\320"
> 2019-03-19 09:49:23,272|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\302\334]Q\354Y;\363\352f\350\304\023^\t\214\211\234l\357\201\230\300)2\245Q\r\365\001\243}"
> 2019-03-19 09:49:23,273|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "@\256\307dY!\364;j\227E\255$3\216{\355r\211_\251_\203t|\030W\nA\234o\221"
> 2019-03-19 09:49:23,273|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: ":\n\262\344H3\034\306\244\036Z\242\004\b\252\264\241\356a\220iy\264\376g\226r\363\200m\204f"
> 2019-03-19 09:49:23,273|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\'\215[mB\363\243\235\002\2731\230e!fP\034\375\006\325\244\031\200\361K\202\020\315*\276\353\355"
> 2019-03-19 09:49:23,274|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "|^\273\\\340#It\263\365=\242\245\\\265\357\206\017\004\346r\306\370=Bbf1F`<\375"
> 2019-03-19 09:49:23,274|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "n\343\233s]oG\"\222\204_E\370\nb5\tPJ\274\232\033\356U\370\324\205\255\250\230\344\201"
> 2019-03-19 09:49:23,274|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\352d\316\363)\303k\305\036\341b\373\261\225:C\a\\\026)\312X\002\305\373\333~\002\306\225\366\231"
> 2019-03-19 09:49:23,275|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\202\355\002\021$!\333t2\336\250TS\243\306D\372\305\310sQ\233+\a\312V3\206\234\2274\370"
> 2019-03-19 09:49:23,275|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\000\206\200gA\314#J\354\016$<\211\v\235^\231\355\211f\257\364\243\327Mn\031\2477\017\367\370"
> 2019-03-19 09:49:23,275|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "5x\212\237\305|]\361~\3336$\"\f+UG\257\232o0\242\027\322\000\252\257\264\207\212\276\275"
> 2019-03-19 09:49:23,275|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:23,276|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:23,276|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:23,276|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|on datanode 1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c
> 2019-03-19 09:49:23,276|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:23,276|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2019-03-19 09:49:23,277|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> 2019-03-19 09:49:23,277|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:247)
> 2019-03-19 09:49:23,277|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:217)
> 2019-03-19 09:49:23,277|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:256)
> 2019-03-19 09:49:23,277|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:292)
> 2019-03-19 09:49:23,278|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.prepareRead(BlockInputStream.java:226)
> 2019-03-19 09:49:23,278|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:150)
> 2019-03-19 09:49:23,278|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.read(KeyInputStream.java:232)
> 2019-03-19 09:49:23,278|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:126)
> 2019-03-19 09:49:23,278|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47)
> 2019-03-19 09:49:23,279|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.io.InputStream.read(InputStream.java:101)
> 2019-03-19 09:49:23,279|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
> 2019-03-19 09:49:23,279|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:98)
> 2019-03-19 09:49:23,279|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:48)
> 2019-03-19 09:49:23,279|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.execute(CommandLine.java:919)
> 2019-03-19 09:49:23,279|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.access$700(CommandLine.java:104)
> 2019-03-19 09:49:23,280|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
> 2019-03-19 09:49:23,280|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
> 2019-03-19 09:49:23,280|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
> 2019-03-19 09:49:23,280|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
> 2019-03-19 09:49:23,281|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
> 2019-03-19 09:49:23,281|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
> 2019-03-19 09:49:23,281|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.execute(Shell.java:84)
> 2019-03-19 09:49:23,281|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
> 2019-03-19 09:49:23,281|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:95)
> 2019-03-19 09:49:23,282|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:23,282|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:526)
> 2019-03-19 09:49:23,282|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434)
> 2019-03-19 09:49:23,282|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:23,283|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:23,283|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:23,283|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678)
> 2019-03-19 09:49:23,284|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:23,284|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:23,284|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:23,285|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
> 2019-03-19 09:49:23,285|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> 2019-03-19 09:49:23,285|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> 2019-03-19 09:49:23,285|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> 2019-03-19 09:49:23,286|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> 2019-03-19 09:49:23,286|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> 2019-03-19 09:49:23,286|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2019-03-19 09:49:23,286|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> 2019-03-19 09:49:23,287|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2019-03-19 09:49:23,287|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2019-03-19 09:49:23,287|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.lang.Thread.run(Thread.java:748)
> 2019-03-19 09:49:23,288|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.27.89.128:9859
> 2019-03-19 09:49:23,288|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 2019-03-19 09:49:23,288|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> 2019-03-19 09:49:23,288|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
> 2019-03-19 09:49:23,289|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
> 2019-03-19 09:49:23,289|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632)
> 2019-03-19 09:49:23,289|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
> 2019-03-19 09:49:23,290|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
> 2019-03-19 09:49:23,290|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> 2019-03-19 09:49:23,290|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> 2019-03-19 09:49:23,291|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2019-03-19 09:49:23,291|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 1 more
> 2019-03-19 09:49:23,291|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: java.net.ConnectException: Connection refused
> 2019-03-19 09:49:23,291|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 11 more
> 2019-03-19 09:49:23,557|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|19/03/19 09:49:23 WARN scm.XceiverClientGrpc: Failed to execute command cmdType: ReadChunk
> 2019-03-19 09:49:23,557|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|traceID: "f5bb707b-b0ef-4c21-a6ea-910b72221cfd"
> 2019-03-19 09:49:23,558|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 1
> 2019-03-19 09:49:23,558|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|datanodeUuid: "1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c"
> 2019-03-19 09:49:23,558|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|readChunk {
> 2019-03-19 09:49:23,559|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockID {
> 2019-03-19 09:49:23,559|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 1
> 2019-03-19 09:49:23,559|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|localID: 101776679626145792
> 2019-03-19 09:49:23,560|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockCommitSequenceId: 23
> 2019-03-19 09:49:23,560|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:23,560|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkData {
> 2019-03-19 09:49:23,561|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkName: "f42c4a33a0b766f6a39981411a77be83_stream_dfa8c5a9-80e8-4c2a-a371-058de7f11f64_chunk_7"
> 2019-03-19 09:49:23,561|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|offset: 0
> 2019-03-19 09:49:23,561|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|len: 16777216
> 2019-03-19 09:49:23,562|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksumData {
> 2019-03-19 09:49:23,562|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|type: SHA256
> 2019-03-19 09:49:23,562|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|bytesPerChecksum: 1048576
> 2019-03-19 09:49:23,562|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:23,563|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "t\353\224_\357\351\372:\377Ik\242\266\004\307~n\245+$\243%.\351x\305\205\315\271\375\352\\"
> 2019-03-19 09:49:23,563|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\327^\207\361~\207\245\321\273\327\a\300.\337\344\"^{U\000\261\334\256|`\223\227\341I\233~\201"
> 2019-03-19 09:49:23,563|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "e|g\357\222]\\\353\203\375\230\032\351\266\274]\331\246\362X1!\035\214\2766\222\2321\235\370f"
> 2019-03-19 09:49:23,564|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\333()a8!l\271\325\273d\v\272\214>!\315\354\245|3S(\001\235\324\221\356\216\372\356\324"
> 2019-03-19 09:49:23,564|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\254\265\254)\232h\224\320\n\261D\352\033[\245\263\3011\211/|5@\367\23485p\227\315jX"
> 2019-03-19 09:49:23,564|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\354\336bw\034\334\221\330\321LQ\317 p\305\265\356\304\317O\237\337|\225+\322\'\t}^=\016"
> 2019-03-19 09:49:23,565|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\204}\217X\236\240Z\001\271a6\234\272:\'\271\000\267\205:\032\"qX\344N\315\214O\277Tu"
> 2019-03-19 09:49:23,565|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\210\307\200\200\027\255\257t\306lw\v\267\216\251\260]\304:\367#\027%R_h&\033\003\233\022F"
> 2019-03-19 09:49:23,565|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: " Dw\331\226\3056\232\017(\266~\243\375\342\272\360\225\217\363\344\351\367\272\0308\340v\001\222\3639"
> 2019-03-19 09:49:23,566|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\270z=\235\030\267L\033-UL\300\357\305\302\2560\260\241\272\307\024\265!\277\346\343U\230\277\317\201"
> 2019-03-19 09:49:23,566|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\332\036:>\004r\361\216\372?y\t\341\026Ik<\241\207\370\000\231\356.\246{7\235\210\360\325\265"
> 2019-03-19 09:49:23,566|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:23,567|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\343\363d\275$]\b\311\232\235wl\036\005\236\216\240\265V\263\350mb\261\261\344;\264D\300\000B"
> 2019-03-19 09:49:23,568|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\363\252\312\233\2252\325\375\215T\222\311\035\005_\245\262\361W\367[\201$\037}\353?m\363/y\224"
> 2019-03-19 09:49:23,568|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "5J\331\364\267\214%\302\341<_\344\333V\214\325mL=\253+\035\215\a\243\310i\v\300\356\250\227"
> 2019-03-19 09:49:23,568|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:23,569|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:23,569|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:23,569|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|on datanode 1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c
> 2019-03-19 09:49:23,570|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:23,570|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2019-03-19 09:49:23,570|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> 2019-03-19 09:49:23,571|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:247)
> 2019-03-19 09:49:23,571|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:217)
> 2019-03-19 09:49:23,571|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:256)
> 2019-03-19 09:49:23,572|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:292)
> 2019-03-19 09:49:23,572|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.prepareRead(BlockInputStream.java:226)
> 2019-03-19 09:49:23,572|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:150)
> 2019-03-19 09:49:23,573|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.read(KeyInputStream.java:232)
> 2019-03-19 09:49:23,573|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:126)
> 2019-03-19 09:49:23,574|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47)
> 2019-03-19 09:49:23,574|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.io.InputStream.read(InputStream.java:101)
> 2019-03-19 09:49:23,574|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
> 2019-03-19 09:49:23,574|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:98)
> 2019-03-19 09:49:23,575|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:48)
> 2019-03-19 09:49:23,575|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.execute(CommandLine.java:919)
> 2019-03-19 09:49:23,575|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.access$700(CommandLine.java:104)
> 2019-03-19 09:49:23,576|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
> 2019-03-19 09:49:23,576|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
> 2019-03-19 09:49:23,576|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
> 2019-03-19 09:49:23,577|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
> 2019-03-19 09:49:23,577|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
> 2019-03-19 09:49:23,577|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
> 2019-03-19 09:49:23,578|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.execute(Shell.java:84)
> 2019-03-19 09:49:23,578|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
> 2019-03-19 09:49:23,578|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:95)
> 2019-03-19 09:49:23,579|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:23,579|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:526)
> 2019-03-19 09:49:23,579|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434)
> 2019-03-19 09:49:23,580|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:23,580|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:23,580|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:23,581|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678)
> 2019-03-19 09:49:23,581|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:23,581|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:23,582|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:23,582|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
> 2019-03-19 09:49:23,582|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> 2019-03-19 09:49:23,583|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> 2019-03-19 09:49:23,583|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> 2019-03-19 09:49:23,583|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> 2019-03-19 09:49:23,584|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> 2019-03-19 09:49:23,584|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2019-03-19 09:49:23,584|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> 2019-03-19 09:49:23,585|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2019-03-19 09:49:23,585|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2019-03-19 09:49:23,585|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.lang.Thread.run(Thread.java:748)
> 2019-03-19 09:49:23,585|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.27.89.128:9859
> 2019-03-19 09:49:23,586|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 2019-03-19 09:49:23,586|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> 2019-03-19 09:49:23,586|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
> 2019-03-19 09:49:23,587|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
> 2019-03-19 09:49:23,587|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632)
> 2019-03-19 09:49:23,587|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
> 2019-03-19 09:49:23,588|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
> 2019-03-19 09:49:23,588|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> 2019-03-19 09:49:23,588|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> 2019-03-19 09:49:23,589|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2019-03-19 09:49:23,589|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 1 more
> 2019-03-19 09:49:23,589|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: java.net.ConnectException: Connection refused
> 2019-03-19 09:49:23,589|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 11 more
> 2019-03-19 09:49:23,798|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|19/03/19 09:49:23 WARN scm.XceiverClientGrpc: Failed to execute command cmdType: ReadChunk
> 2019-03-19 09:49:23,798|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|traceID: "f5bb707b-b0ef-4c21-a6ea-910b72221cfd"
> 2019-03-19 09:49:23,799|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 1
> 2019-03-19 09:49:23,799|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|datanodeUuid: "1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c"
> 2019-03-19 09:49:23,799|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|readChunk {
> 2019-03-19 09:49:23,800|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockID {
> 2019-03-19 09:49:23,800|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 1
> 2019-03-19 09:49:23,800|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|localID: 101776679626145792
> 2019-03-19 09:49:23,801|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockCommitSequenceId: 23
> 2019-03-19 09:49:23,801|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:23,801|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkData {
> 2019-03-19 09:49:23,801|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkName: "f42c4a33a0b766f6a39981411a77be83_stream_dfa8c5a9-80e8-4c2a-a371-058de7f11f64_chunk_8"
> 2019-03-19 09:49:23,802|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|offset: 0
> 2019-03-19 09:49:23,802|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|len: 16777216
> 2019-03-19 09:49:23,802|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksumData {
> 2019-03-19 09:49:23,803|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|type: SHA256
> 2019-03-19 09:49:23,803|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|bytesPerChecksum: 1048576
> 2019-03-19 09:49:23,804|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:23,804|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "H\304\304dy\351=\350\307F\224\275\256\026&\300\244\233\302j\301)H\303MCW\031\336]CI"
> 2019-03-19 09:49:23,804|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\207\\\277\333\032[N\227\234\314\356\316\340\330\032\320\303l\037\333\001\305\345\342\364l\345\003\021;?P"
> 2019-03-19 09:49:23,804|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "x\323\031\317\214\237z\304g>\356\375\374{\301\324}y\035\373\325\266\313\003\223\301\231\317\232a\236\212"
> 2019-03-19 09:49:23,805|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "z\026\374HI\373Y$Q\360\271T\253\276\241\276h/\370L\3527D\345\004\2767\256\340\364\020\230"
> 2019-03-19 09:49:23,805|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\r\345n\363\005\017C\311\036M\304p\272\330\302z\354]8m\324\025}\207\330\026\315\3768\361l9"
> 2019-03-19 09:49:23,806|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\3516\312\350\260\177eU\244\020\'\022rT\034\320\363\3663\332\322\204e\235\225\356J\332\330L\303\253"
> 2019-03-19 09:49:23,806|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\311\241\255\354\261\244\036\003\366\354y\321b\2501\331\fc.\273\214c8\r\377*\254,f\327\275/"
> 2019-03-19 09:49:23,806|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\376!\276\341Y\312\374\252\003w;\215Sx\030\\G>\2521\362\325A\027\034\315\361:\251\034R\321"
> 2019-03-19 09:49:23,807|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\3323\301+\034<\341\226\264\251\330\255\000\261\037\232\226y\264\257^\357^\006\262\000\200\200\033tt\346"
> 2019-03-19 09:49:23,807|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\373\330\250o\374b\274iD\332\270\002\005\216T\032\271\005}ju\357\031\264L\233\f\335\267\b\335\203"
> 2019-03-19 09:49:23,808|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\0233a\246\305e\265\326\243\270y1z\226\250#\215\032\305\260Z\354o\211\027\327G\v\346X\202\022"
> 2019-03-19 09:49:23,808|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\212>\2455\231\270yU\255Dv\320_\320\330A&8\243\330d\027@\362\024\241\004E\212\255!\351"
> 2019-03-19 09:49:23,808|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "L\n\205N\256#\t\327j\366\023B.\2139\303\225e+\255wD\357\326w\'\362C\270\035\034\240"
> 2019-03-19 09:49:23,809|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "e\310U\374\265\201\257 ?q!\022\364P9\244B\"\n\326\352J\263Y\034Q\3460\tt\261F"
> 2019-03-19 09:49:23,809|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "B\177\r-\004\353$=[c\037\n\302E\034\335\365\321]\302m\370 \027\320\207\233-\375q\226^"
> 2019-03-19 09:49:23,809|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:23,810|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:23,810|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:23,810|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|on datanode 1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c
> 2019-03-19 09:49:23,811|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:23,811|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2019-03-19 09:49:23,811|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> 2019-03-19 09:49:23,811|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:247)
> 2019-03-19 09:49:23,812|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:217)
> 2019-03-19 09:49:23,812|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:256)
> 2019-03-19 09:49:23,812|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:292)
> 2019-03-19 09:49:23,812|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.prepareRead(BlockInputStream.java:226)
> 2019-03-19 09:49:23,812|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:150)
> 2019-03-19 09:49:23,813|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.read(KeyInputStream.java:232)
> 2019-03-19 09:49:23,813|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:126)
> 2019-03-19 09:49:23,813|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47)
> 2019-03-19 09:49:23,813|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.io.InputStream.read(InputStream.java:101)
> 2019-03-19 09:49:23,814|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
> 2019-03-19 09:49:23,814|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:98)
> 2019-03-19 09:49:23,814|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:48)
> 2019-03-19 09:49:23,814|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.execute(CommandLine.java:919)
> 2019-03-19 09:49:23,815|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.access$700(CommandLine.java:104)
> 2019-03-19 09:49:23,815|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
> 2019-03-19 09:49:23,815|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
> 2019-03-19 09:49:23,815|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
> 2019-03-19 09:49:23,816|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
> 2019-03-19 09:49:23,816|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
> 2019-03-19 09:49:23,816|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
> 2019-03-19 09:49:23,816|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.execute(Shell.java:84)
> 2019-03-19 09:49:23,817|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
> 2019-03-19 09:49:23,817|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:95)
> 2019-03-19 09:49:23,817|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:23,817|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:526)
> 2019-03-19 09:49:23,818|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434)
> 2019-03-19 09:49:23,818|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:23,818|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:23,818|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:23,819|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678)
> 2019-03-19 09:49:23,819|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:23,819|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:23,820|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:23,820|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
> 2019-03-19 09:49:23,820|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> 2019-03-19 09:49:23,820|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> 2019-03-19 09:49:23,821|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> 2019-03-19 09:49:23,821|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> 2019-03-19 09:49:23,821|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> 2019-03-19 09:49:23,821|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2019-03-19 09:49:23,821|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> 2019-03-19 09:49:23,822|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2019-03-19 09:49:23,822|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2019-03-19 09:49:23,822|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.lang.Thread.run(Thread.java:748)
> 2019-03-19 09:49:23,822|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.27.89.128:9859
> 2019-03-19 09:49:23,823|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 2019-03-19 09:49:23,823|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> 2019-03-19 09:49:23,823|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
> 2019-03-19 09:49:23,823|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
> 2019-03-19 09:49:23,823|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632)
> 2019-03-19 09:49:23,824|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
> 2019-03-19 09:49:23,824|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
> 2019-03-19 09:49:23,824|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> 2019-03-19 09:49:23,825|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> 2019-03-19 09:49:23,825|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2019-03-19 09:49:23,825|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 1 more
> 2019-03-19 09:49:23,825|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: java.net.ConnectException: Connection refused
> 2019-03-19 09:49:23,825|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 11 more
> 2019-03-19 09:49:24,064|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|19/03/19 09:49:23 WARN scm.XceiverClientGrpc: Failed to execute command cmdType: ReadChunk
> 2019-03-19 09:49:24,064|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|traceID: "f5bb707b-b0ef-4c21-a6ea-910b72221cfd"
> 2019-03-19 09:49:24,064|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 2
> 2019-03-19 09:49:24,065|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|datanodeUuid: "1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c"
> 2019-03-19 09:49:24,066|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|readChunk {
> 2019-03-19 09:49:24,066|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockID {
> 2019-03-19 09:49:24,066|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 2
> 2019-03-19 09:49:24,067|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|localID: 101776681938518017
> 2019-03-19 09:49:24,067|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockCommitSequenceId: 43
> 2019-03-19 09:49:24,068|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:24,068|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkData {
> 2019-03-19 09:49:24,068|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkName: "f42c4a33a0b766f6a39981411a77be83_stream_fce2ae16-39ce-4f43-bf29-3db715ea2ced_chunk_1"
> 2019-03-19 09:49:24,069|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|offset: 0
> 2019-03-19 09:49:24,069|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|len: 16777216
> 2019-03-19 09:49:24,069|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksumData {
> 2019-03-19 09:49:24,070|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|type: SHA256
> 2019-03-19 09:49:24,070|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|bytesPerChecksum: 1048576
> 2019-03-19 09:49:24,071|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\321\002n\027\255\307,\242\232\275\365*\266\"\v\370\020\275\334\357\301d\2259\300*\353\254\2711\032\030"
> 2019-03-19 09:49:24,071|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "n?\375\351=l0+\276\037\033\227J\274\255\005\357\335\300yG\346\324\323<Y\312\003\330\\y\275"
> 2019-03-19 09:49:24,071|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\034\226M)#1*\324\233i\310\303\345cP\267jU\211\f\266\216*Q\345\340\262\312q\211\3122"
> 2019-03-19 09:49:24,072|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\312\302\360\177Y\346\031\223\351Ks\212u\330p\202=\204p\032\230\000GA\272R\307\2662\235Q\276"
> 2019-03-19 09:49:24,072|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:24,073|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:24,073|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:24,074|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\321\372G\203_\256nbB\256w\334\201C\342/\006\234\263i\357\220\252\370\"vEB.od\232"
> 2019-03-19 09:49:24,074|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "O\342nb\330\303?\005\354\264\347\003\246\377\205\t\313\336\235\306\232\017*\244\353\036\211\270\b8J\024"
> 2019-03-19 09:49:24,074|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:24,075|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:24,075|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\350\325 p\342\272f\200\313\257\377y\223<6$\2170\274\036T\332a\b\r\215`\373\341@\262X"
> 2019-03-19 09:49:24,076|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "*n\320\f<\347OV\316\323@\351rx\211\037\330\242j/V\205\352\340\376 \3076\333\004\033\365"
> 2019-03-19 09:49:24,076|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\v\262\036\025t6\257\0250\367\a\315\235\233\211&\344\231\336\325B\345\246,\313EX\265\240=v#"
> 2019-03-19 09:49:24,077|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:24,077|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:24,077|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:24,077|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:24,078|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:24,078|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|on datanode 1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c
> 2019-03-19 09:49:24,079|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:24,079|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2019-03-19 09:49:24,079|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> 2019-03-19 09:49:24,080|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:247)
> 2019-03-19 09:49:24,080|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:217)
> 2019-03-19 09:49:24,081|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:256)
> 2019-03-19 09:49:24,081|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:292)
> 2019-03-19 09:49:24,081|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.prepareRead(BlockInputStream.java:226)
> 2019-03-19 09:49:24,082|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:150)
> 2019-03-19 09:49:24,082|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.read(KeyInputStream.java:232)
> 2019-03-19 09:49:24,083|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:126)
> 2019-03-19 09:49:24,083|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47)
> 2019-03-19 09:49:24,083|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.io.InputStream.read(InputStream.java:101)
> 2019-03-19 09:49:24,084|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
> 2019-03-19 09:49:24,084|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:98)
> 2019-03-19 09:49:24,085|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:48)
> 2019-03-19 09:49:24,085|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.execute(CommandLine.java:919)
> 2019-03-19 09:49:24,085|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.access$700(CommandLine.java:104)
> 2019-03-19 09:49:24,086|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
> 2019-03-19 09:49:24,086|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
> 2019-03-19 09:49:24,086|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
> 2019-03-19 09:49:24,087|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
> 2019-03-19 09:49:24,087|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
> 2019-03-19 09:49:24,087|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
> 2019-03-19 09:49:24,088|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.execute(Shell.java:84)
> 2019-03-19 09:49:24,088|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
> 2019-03-19 09:49:24,088|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:95)
> 2019-03-19 09:49:24,089|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:24,089|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:526)
> 2019-03-19 09:49:24,090|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434)
> 2019-03-19 09:49:24,090|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:24,090|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:24,091|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:24,091|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678)
> 2019-03-19 09:49:24,092|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:24,092|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:24,092|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:24,093|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
> 2019-03-19 09:49:24,093|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> 2019-03-19 09:49:24,093|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> 2019-03-19 09:49:24,094|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> 2019-03-19 09:49:24,094|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> 2019-03-19 09:49:24,095|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> 2019-03-19 09:49:24,095|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2019-03-19 09:49:24,095|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> 2019-03-19 09:49:24,096|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2019-03-19 09:49:24,096|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2019-03-19 09:49:24,096|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.lang.Thread.run(Thread.java:748)
> 2019-03-19 09:49:24,097|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.27.89.128:9859
> 2019-03-19 09:49:24,097|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 2019-03-19 09:49:24,097|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> 2019-03-19 09:49:24,098|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
> 2019-03-19 09:49:24,098|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
> 2019-03-19 09:49:24,098|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632)
> 2019-03-19 09:49:24,099|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
> 2019-03-19 09:49:24,099|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
> 2019-03-19 09:49:24,099|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> 2019-03-19 09:49:24,100|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> 2019-03-19 09:49:24,100|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2019-03-19 09:49:24,100|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 1 more
> 2019-03-19 09:49:24,101|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: java.net.ConnectException: Connection refused
> 2019-03-19 09:49:24,101|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 11 more
> 2019-03-19 09:49:24,271|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|19/03/19 09:49:24 WARN scm.XceiverClientGrpc: Failed to execute command cmdType: ReadChunk
> 2019-03-19 09:49:24,271|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|traceID: "f5bb707b-b0ef-4c21-a6ea-910b72221cfd"
> 2019-03-19 09:49:24,271|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 2
> 2019-03-19 09:49:24,272|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|datanodeUuid: "1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c"
> 2019-03-19 09:49:24,272|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|readChunk {
> 2019-03-19 09:49:24,272|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockID {
> 2019-03-19 09:49:24,273|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 2
> 2019-03-19 09:49:24,273|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|localID: 101776681938518017
> 2019-03-19 09:49:24,273|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockCommitSequenceId: 43
> 2019-03-19 09:49:24,273|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:24,273|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkData {
> 2019-03-19 09:49:24,274|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkName: "f42c4a33a0b766f6a39981411a77be83_stream_fce2ae16-39ce-4f43-bf29-3db715ea2ced_chunk_2"
> 2019-03-19 09:49:24,274|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|offset: 0
> 2019-03-19 09:49:24,274|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|len: 16777216
> 2019-03-19 09:49:24,274|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksumData {
> 2019-03-19 09:49:24,274|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|type: SHA256
> 2019-03-19 09:49:24,274|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|bytesPerChecksum: 1048576
> 2019-03-19 09:49:24,275|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:24,275|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:24,275|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:24,276|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "3\365V\301\305\035\337\3058\361\031\203\3604\034\305{\253\252b\2055\261\223N5sYA\025\271\242"
> 2019-03-19 09:49:24,276|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\002(\320\245\262\226\233G\211\356\331\324EU%\027\276\245\335\327\267+\241\352\314D@\205\271Fu&"
> 2019-03-19 09:49:24,276|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\3106\260\201\017\362\360\317\363\264g\271\347f\354C\343m\234\026\230\303\274\t\372\356\370\204\006\326]\345"
> 2019-03-19 09:49:24,276|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\343\212\330W\255\222w@\261\322Y\307\247\337\017\321\366\b\000\213\233\234\245Us<\nW\274\005\373\341"
> 2019-03-19 09:49:24,277|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\000\342\352\262,|\302\026\275\302\374}\324\264\354\'!\271\371l\350\304\320K\003\300\225\023\264\251\246\274"
> 2019-03-19 09:49:24,277|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "L\352[4U\252y\310c\334s/\366\361\r$\203\225\357\202\210\343\273\340\024s\022k\313\336S\261"
> 2019-03-19 09:49:24,277|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\200\322\f\272\aF\266\0056\212\270W\002\354\257\204\350\312\257_\206Fi\265E*<U\307Ze\023"
> 2019-03-19 09:49:24,277|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\b\354\277.\374\3771Nn$\277#\202j\324CY\337\vC\231\245Z\250\037OO\365\317\207\177\212"
> 2019-03-19 09:49:24,278|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\3474Q\025\353.\205\315[\221\216\276\344:\325\227\265\331q\355\356?\230}\334QQ|\314\215\347\321"
> 2019-03-19 09:49:24,278|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\037\\\3536U\360\333\252/^T\357\267#\237)8\363S\347v\034S\354c\207\233\303\366\336\026z"
> 2019-03-19 09:49:24,278|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\n\205\367\256>\205\242\331\234Fn\203.\304%\333\022\236\367\201AN\336\227\207+m\367\231\226-\304"
> 2019-03-19 09:49:24,278|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\200ztd\211\276\2479\356^\316\036^\370\326:\206\306\a[<\317\002\331\376[\300\354l\"\231\\"
> 2019-03-19 09:49:24,279|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\272z\264\275\000\021\236A\021\336&\344z\t\263\002\035`\200\250p%\322$\263\276\n\301\345\"A\241"
> 2019-03-19 09:49:24,279|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:24,279|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:24,279|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:24,279|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|on datanode 1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c
> 2019-03-19 09:49:24,280|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:24,280|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2019-03-19 09:49:24,280|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> 2019-03-19 09:49:24,280|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:247)
> 2019-03-19 09:49:24,280|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:217)
> 2019-03-19 09:49:24,281|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:256)
> 2019-03-19 09:49:24,281|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:292)
> 2019-03-19 09:49:24,281|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.prepareRead(BlockInputStream.java:226)
> 2019-03-19 09:49:24,282|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:150)
> 2019-03-19 09:49:24,282|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.read(KeyInputStream.java:232)
> 2019-03-19 09:49:24,282|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:126)
> 2019-03-19 09:49:24,282|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47)
> 2019-03-19 09:49:24,283|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.io.InputStream.read(InputStream.java:101)
> 2019-03-19 09:49:24,283|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
> 2019-03-19 09:49:24,283|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:98)
> 2019-03-19 09:49:24,284|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:48)
> 2019-03-19 09:49:24,284|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.execute(CommandLine.java:919)
> 2019-03-19 09:49:24,284|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.access$700(CommandLine.java:104)
> 2019-03-19 09:49:24,284|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
> 2019-03-19 09:49:24,284|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
> 2019-03-19 09:49:24,285|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
> 2019-03-19 09:49:24,285|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
> 2019-03-19 09:49:24,285|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
> 2019-03-19 09:49:24,285|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
> 2019-03-19 09:49:24,286|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.execute(Shell.java:84)
> 2019-03-19 09:49:24,286|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
> 2019-03-19 09:49:24,286|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:95)
> 2019-03-19 09:49:24,287|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:24,287|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:526)
> 2019-03-19 09:49:24,287|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434)
> 2019-03-19 09:49:24,287|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:24,288|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:24,288|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:24,288|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678)
> 2019-03-19 09:49:24,288|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:24,289|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:24,289|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:24,289|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
> 2019-03-19 09:49:24,289|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> 2019-03-19 09:49:24,290|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> 2019-03-19 09:49:24,290|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> 2019-03-19 09:49:24,290|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> 2019-03-19 09:49:24,290|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> 2019-03-19 09:49:24,291|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2019-03-19 09:49:24,291|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> 2019-03-19 09:49:24,291|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2019-03-19 09:49:24,291|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2019-03-19 09:49:24,291|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.lang.Thread.run(Thread.java:748)
> 2019-03-19 09:49:24,292|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.27.89.128:9859
> 2019-03-19 09:49:24,292|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 2019-03-19 09:49:24,292|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> 2019-03-19 09:49:24,292|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
> 2019-03-19 09:49:24,292|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
> 2019-03-19 09:49:24,293|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632)
> 2019-03-19 09:49:24,293|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
> 2019-03-19 09:49:24,293|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
> 2019-03-19 09:49:24,293|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> 2019-03-19 09:49:24,293|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> 2019-03-19 09:49:24,294|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2019-03-19 09:49:24,294|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 1 more
> 2019-03-19 09:49:24,294|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: java.net.ConnectException: Connection refused
> 2019-03-19 09:49:24,294|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 11 more
> 2019-03-19 09:49:24,682|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|19/03/19 09:49:24 WARN scm.XceiverClientGrpc: Failed to execute command cmdType: ReadChunk
> 2019-03-19 09:49:24,683|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|traceID: "f5bb707b-b0ef-4c21-a6ea-910b72221cfd"
> 2019-03-19 09:49:24,683|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 2
> 2019-03-19 09:49:24,684|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|datanodeUuid: "1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c"
> 2019-03-19 09:49:24,684|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|readChunk {
> 2019-03-19 09:49:24,684|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockID {
> 2019-03-19 09:49:24,685|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 2
> 2019-03-19 09:49:24,685|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|localID: 101776681938518017
> 2019-03-19 09:49:24,685|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockCommitSequenceId: 43
> 2019-03-19 09:49:24,686|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:24,686|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkData {
> 2019-03-19 09:49:24,686|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkName: "f42c4a33a0b766f6a39981411a77be83_stream_fce2ae16-39ce-4f43-bf29-3db715ea2ced_chunk_3"
> 2019-03-19 09:49:24,687|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|offset: 0
> 2019-03-19 09:49:24,687|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|len: 16777216
> 2019-03-19 09:49:24,688|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksumData {
> 2019-03-19 09:49:24,688|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|type: SHA256
> 2019-03-19 09:49:24,688|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|bytesPerChecksum: 1048576
> 2019-03-19 09:49:24,688|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\366\200\253\212$(?]r\303\321P\273JxX\273\023\333R\223\230\374D\270\203\263i\254\017\027;"
> 2019-03-19 09:49:24,689|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\f\200\241\216\261:6\334\037!*\314\231\r)\000\314s\207W1\266\202v\315\003\235\210\232\365|1"
> 2019-03-19 09:49:24,689|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "l\313\352?\304\234\374i\255u\032\343\022\v\377\245\177\317\200\343[\270\203Gb\200\315\204\366~\376\026"
> 2019-03-19 09:49:24,690|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\021*~\004\247P\231\216\206q\256\323\343p\365\207\031\001\321@\224\335\240\"\221bUw\2546-9"
> 2019-03-19 09:49:24,690|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "!c\037?\300\242\034\265\210D8\245\200\225\216W\'\304\370\230\263\027q\204f9\204+\274\356R\267"
> 2019-03-19 09:49:24,690|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\256J\335\370\331^\211\274]\223\372]\314\340\345\254bli\225\232\222`\020\326O9\255X\341\333s"
> 2019-03-19 09:49:24,691|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\231\231\375\317\335\246\212\243\333\272K\0054\340\0377l\324\0164OK\363|:k\334Q\024\232\005R"
> 2019-03-19 09:49:24,691|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\"Q\343&\366?\231<\226,\245\235,\241p\367!\003\246\277\244\362\252,\330o\277\210\344B\262\354"
> 2019-03-19 09:49:24,691|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\234\001\263\251#\023\321-\2779\311/\341U-~g\306\271\000\233\201L\340\245\245XE\\\335\244\330"
> 2019-03-19 09:49:24,692|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\365;\200\022\037=\rmHl\0035\021C\236f\300amTsU\244\241\301[\346\026K7\300\373"
> 2019-03-19 09:49:24,692|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\031\241\021&\240\305\210\215\365=\277u\220U\n \274?P\275\320\aDB\001\321\363Gb\262\232!"
> 2019-03-19 09:49:24,692|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\361n=\343\3720\212`/\026\371\217\240\301GZ\341`\235\315\256\342(\337\032\215X@J3\275U"
> 2019-03-19 09:49:24,693|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "^\344\204\354\3018\355 U;\335\230\271\234\377\257K\262\005\250\025K\267_g\363\002\313\332\236JB"
> 2019-03-19 09:49:24,693|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\032\224o\351\211\261y\260\303\234\261?\031Rza6D\202\024\335m\272e\331x\177r\212`\'\342"
> 2019-03-19 09:49:24,694|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "V\316\227\033\255\355\351\304d\2608m\273\213\247\216h1\222\212\215\230\2745B\204\230`\024\272C\024"
> 2019-03-19 09:49:24,694|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\037\220\376\a\235;\236\036`0\217\354\327d(\267Wp\324d\313\225=\261hu-L\337j46"
> 2019-03-19 09:49:24,694|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:24,694|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:24,694|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:24,695|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|on datanode 1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c
> 2019-03-19 09:49:24,695|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:24,695|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2019-03-19 09:49:24,695|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> 2019-03-19 09:49:24,696|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:247)
> 2019-03-19 09:49:24,696|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:217)
> 2019-03-19 09:49:24,696|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:256)
> 2019-03-19 09:49:24,696|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:292)
> 2019-03-19 09:49:24,697|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.prepareRead(BlockInputStream.java:226)
> 2019-03-19 09:49:24,697|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:150)
> 2019-03-19 09:49:24,697|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.read(KeyInputStream.java:232)
> 2019-03-19 09:49:24,697|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:126)
> 2019-03-19 09:49:24,698|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47)
> 2019-03-19 09:49:24,698|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.io.InputStream.read(InputStream.java:101)
> 2019-03-19 09:49:24,698|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
> 2019-03-19 09:49:24,698|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:98)
> 2019-03-19 09:49:24,698|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:48)
> 2019-03-19 09:49:24,699|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.execute(CommandLine.java:919)
> 2019-03-19 09:49:24,699|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.access$700(CommandLine.java:104)
> 2019-03-19 09:49:24,699|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
> 2019-03-19 09:49:24,699|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
> 2019-03-19 09:49:24,699|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
> 2019-03-19 09:49:24,700|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
> 2019-03-19 09:49:24,700|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
> 2019-03-19 09:49:24,700|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
> 2019-03-19 09:49:24,700|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.execute(Shell.java:84)
> 2019-03-19 09:49:24,701|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
> 2019-03-19 09:49:24,701|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:95)
> 2019-03-19 09:49:24,701|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:24,701|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:526)
> 2019-03-19 09:49:24,702|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434)
> 2019-03-19 09:49:24,702|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:24,702|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:24,702|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:24,703|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678)
> 2019-03-19 09:49:24,703|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:24,703|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:24,703|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:24,703|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
> 2019-03-19 09:49:24,704|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> 2019-03-19 09:49:24,704|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> 2019-03-19 09:49:24,704|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> 2019-03-19 09:49:24,704|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> 2019-03-19 09:49:24,705|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> 2019-03-19 09:49:24,705|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2019-03-19 09:49:24,705|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> 2019-03-19 09:49:24,705|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2019-03-19 09:49:24,705|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2019-03-19 09:49:24,705|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.lang.Thread.run(Thread.java:748)
> 2019-03-19 09:49:24,706|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.27.89.128:9859
> 2019-03-19 09:49:24,706|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 2019-03-19 09:49:24,706|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> 2019-03-19 09:49:24,706|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
> 2019-03-19 09:49:24,707|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
> 2019-03-19 09:49:24,707|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632)
> 2019-03-19 09:49:24,707|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
> 2019-03-19 09:49:24,707|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
> 2019-03-19 09:49:24,708|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> 2019-03-19 09:49:24,708|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> 2019-03-19 09:49:24,708|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2019-03-19 09:49:24,708|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 1 more
> 2019-03-19 09:49:24,708|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: java.net.ConnectException: Connection refused
> 2019-03-19 09:49:24,708|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 11 more
> 2019-03-19 09:49:24,876|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|19/03/19 09:49:24 WARN scm.XceiverClientGrpc: Failed to execute command cmdType: ReadChunk
> 2019-03-19 09:49:24,877|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|traceID: "f5bb707b-b0ef-4c21-a6ea-910b72221cfd"
> 2019-03-19 09:49:24,877|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 2
> 2019-03-19 09:49:24,878|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|datanodeUuid: "1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c"
> 2019-03-19 09:49:24,878|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|readChunk {
> 2019-03-19 09:49:24,879|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockID {
> 2019-03-19 09:49:24,879|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 2
> 2019-03-19 09:49:24,879|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|localID: 101776681938518017
> 2019-03-19 09:49:24,879|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockCommitSequenceId: 43
> 2019-03-19 09:49:24,880|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:24,880|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkData {
> 2019-03-19 09:49:24,880|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkName: "f42c4a33a0b766f6a39981411a77be83_stream_fce2ae16-39ce-4f43-bf29-3db715ea2ced_chunk_4"
> 2019-03-19 09:49:24,881|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|offset: 0
> 2019-03-19 09:49:24,881|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|len: 16777216
> 2019-03-19 09:49:24,881|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksumData {
> 2019-03-19 09:49:24,881|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|type: SHA256
> 2019-03-19 09:49:24,882|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|bytesPerChecksum: 1048576
> 2019-03-19 09:49:24,882|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "Q&(\2329a\031\363\350\ro`NHL\177\302\252\220\233.a4\234X\313\214\317\345vl\264"
> 2019-03-19 09:49:24,882|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\\2\027\327\375=\353\304\257\345\345n\364\325\023\367\203\301%\250\035\005n\315\245\3736\005]\226\311G"
> 2019-03-19 09:49:24,882|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\016l\353\022&\023\232\0001\031\242\225E\006\356\262\370J)/\2108J\370\006_\264\2131i\006\022"
> 2019-03-19 09:49:24,883|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\307\374\005\231\306\376\347\r\027\253\352\305\275\342\267\"\f\300\177\206\241Z\361\256\324VIi#\337\035."
> 2019-03-19 09:49:24,883|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "p\341\333\330?X\247\213\206\257l\320v\315?\262\241\277`H\347\320E\205\033\215\211geu\333="
> 2019-03-19 09:49:24,883|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\027\vz\004\247%\222\377\362\354\316`\333\377\241b\347\273j\"\301\357\206\342\321\b\331\027\227\223\264\365"
> 2019-03-19 09:49:24,883|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\254\335\316\277@h\334L\232\332\376\355\370\330f\247\037\003-\005$\201\247v\323IE\261ua\263\027"
> 2019-03-19 09:49:24,884|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\2517\354\226\366\025\317H|.z\221\375\266\300\251\220+WvTd\366\300\331\310\217O\021[\213\362"
> 2019-03-19 09:49:24,884|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\265\247\301\202\361\303\346b<\000\034X\311L\211wSS\342 \217+\227\273\353I\345\361M\017\322\312"
> 2019-03-19 09:49:24,884|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\314\315\230\263-\333sB\225\274Dd\321J\367\240\245[\236\347W\373B\305\326\v\343lR;\371&"
> 2019-03-19 09:49:24,884|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\b\261\345{s\266\353F\366\362\005}\300[\235\245\224\313z\267\005K\023\304\fK%Jq\346\266Z"
> 2019-03-19 09:49:24,885|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "X&w?Rnh\342\266$JK\355\234\232\200\030\227\236]\227\'B\240\334\rx\331.\344k\270"
> 2019-03-19 09:49:24,885|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\277\357\200\326(\201D\271wi\227\320\002\\\330\2438c\325\a~\311\364\f\201\\p\326\274Y\263\255"
> 2019-03-19 09:49:24,885|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\272\200\213\324\005 \267cd\357G\227\245B\\-a#\362V\264\315\310?\350\032\316\032\036@\226P"
> 2019-03-19 09:49:24,885|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\001\360&\272;\302#\246\301v\266:\0212F\314\341\322$\326\326A\212\3621\370f\021\231\'\261T"
> 2019-03-19 09:49:24,886|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\021|S\323\000T~HL\230\005j\370 \371\325\216\231\366lk\322\350\347\263\267\211\317U\250\200\213"
> 2019-03-19 09:49:24,886|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:24,886|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:24,886|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:24,886|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|on datanode 1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c
> 2019-03-19 09:49:24,886|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:24,887|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2019-03-19 09:49:24,887|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> 2019-03-19 09:49:24,887|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:247)
> 2019-03-19 09:49:24,887|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:217)
> 2019-03-19 09:49:24,888|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:256)
> 2019-03-19 09:49:24,888|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:292)
> 2019-03-19 09:49:24,888|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.prepareRead(BlockInputStream.java:226)
> 2019-03-19 09:49:24,888|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:150)
> 2019-03-19 09:49:24,889|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.read(KeyInputStream.java:232)
> 2019-03-19 09:49:24,889|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:126)
> 2019-03-19 09:49:24,889|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47)
> 2019-03-19 09:49:24,889|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.io.InputStream.read(InputStream.java:101)
> 2019-03-19 09:49:24,889|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
> 2019-03-19 09:49:24,890|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:98)
> 2019-03-19 09:49:24,890|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:48)
> 2019-03-19 09:49:24,890|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.execute(CommandLine.java:919)
> 2019-03-19 09:49:24,890|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.access$700(CommandLine.java:104)
> 2019-03-19 09:49:24,890|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
> 2019-03-19 09:49:24,891|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
> 2019-03-19 09:49:24,891|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
> 2019-03-19 09:49:24,891|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
> 2019-03-19 09:49:24,891|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
> 2019-03-19 09:49:24,892|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
> 2019-03-19 09:49:24,892|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.execute(Shell.java:84)
> 2019-03-19 09:49:24,892|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
> 2019-03-19 09:49:24,892|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:95)
> 2019-03-19 09:49:24,892|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:24,892|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:526)
> 2019-03-19 09:49:24,893|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434)
> 2019-03-19 09:49:24,893|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:24,893|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:24,893|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:24,894|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678)
> 2019-03-19 09:49:24,894|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:24,894|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:24,894|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:24,895|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
> 2019-03-19 09:49:24,895|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> 2019-03-19 09:49:24,895|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> 2019-03-19 09:49:24,896|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> 2019-03-19 09:49:24,896|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> 2019-03-19 09:49:24,896|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> 2019-03-19 09:49:24,896|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2019-03-19 09:49:24,897|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> 2019-03-19 09:49:24,897|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2019-03-19 09:49:24,897|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2019-03-19 09:49:24,897|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.lang.Thread.run(Thread.java:748)
> 2019-03-19 09:49:24,897|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.27.89.128:9859
> 2019-03-19 09:49:24,898|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 2019-03-19 09:49:24,898|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> 2019-03-19 09:49:24,898|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
> 2019-03-19 09:49:24,898|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
> 2019-03-19 09:49:24,899|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632)
> 2019-03-19 09:49:24,899|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
> 2019-03-19 09:49:24,899|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
> 2019-03-19 09:49:24,899|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> 2019-03-19 09:49:24,899|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> 2019-03-19 09:49:24,900|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2019-03-19 09:49:24,900|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 1 more
> 2019-03-19 09:49:24,900|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: java.net.ConnectException: Connection refused
> 2019-03-19 09:49:24,900|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 11 more
> 2019-03-19 09:49:25,173|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|19/03/19 09:49:25 WARN scm.XceiverClientGrpc: Failed to execute command cmdType: ReadChunk
> 2019-03-19 09:49:25,173|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|traceID: "f5bb707b-b0ef-4c21-a6ea-910b72221cfd"
> 2019-03-19 09:49:25,173|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 2
> 2019-03-19 09:49:25,174|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|datanodeUuid: "1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c"
> 2019-03-19 09:49:25,174|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|readChunk {
> 2019-03-19 09:49:25,174|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockID {
> 2019-03-19 09:49:25,174|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 2
> 2019-03-19 09:49:25,175|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|localID: 101776681938518017
> 2019-03-19 09:49:25,175|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockCommitSequenceId: 43
> 2019-03-19 09:49:25,175|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:25,175|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkData {
> 2019-03-19 09:49:25,176|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkName: "f42c4a33a0b766f6a39981411a77be83_stream_fce2ae16-39ce-4f43-bf29-3db715ea2ced_chunk_5"
> 2019-03-19 09:49:25,176|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|offset: 0
> 2019-03-19 09:49:25,176|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|len: 16777216
> 2019-03-19 09:49:25,176|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksumData {
> 2019-03-19 09:49:25,176|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|type: SHA256
> 2019-03-19 09:49:25,176|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|bytesPerChecksum: 1048576
> 2019-03-19 09:49:25,177|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: ",\302\247\203\372>Q\031\272\322L\233\327A\022\322\371N\000\365VM\263g\272\rLo\32703C"
> 2019-03-19 09:49:25,177|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\216=n\245\242\3274\025\306\3576\341u\034\240\016`7\225:5\272\267\\\251Pz\202\1772\357\215"
> 2019-03-19 09:49:25,177|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\201\347\342\320V\351\207\252\362,\022/>\a\f\222\341\355\234\331\177[c\356\"\356\366\253\017\'+<"
> 2019-03-19 09:49:25,178|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\373\312d\355M\016\276+w\321\3635]\364\217\265/\371\274\333V]\347\367\001\330\246`\371\036\037\221"
> 2019-03-19 09:49:25,178|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\334\346\271#\343\034\222\241x\270\272\327\330\253\331\224,\235\035\\\005\177\360\354\334p\353\342\271\036\360\020"
> 2019-03-19 09:49:25,178|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\326\004\331\233\233\272f\366P+7\305\334\'\032\032\003\355I\020\231E=\277\331)\212\202*`\371\351"
> 2019-03-19 09:49:25,178|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\270\240^\273\327\245\245\311\025\233\362\242\355\376\016P\332\313\272\260\234\211\a\2724\203\273\036\346?H\031"
> 2019-03-19 09:49:25,179|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\003\256/\257\316\352\033v\265\'\027\037\354O\377\211\240\220\320\024\247\236}u\031q1`\353\204\277\230"
> 2019-03-19 09:49:25,179|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\020\340Rl\vk\316\021y\316\006\b\027R\312\243<;\255\356\245hi\351\314\354\tC,!\246\321"
> 2019-03-19 09:49:25,179|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\275\246\003h\317\207\370\271\313\225.\2014p\335\307\t\322,\'\221\357\345F\300>\302\365\377\214\033\332"
> 2019-03-19 09:49:25,180|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\333\201\205\207\366oL\335e\0204\003\0208\355\207T\211\206\305\232Z\316\207\274[P\340e\246\254\270"
> 2019-03-19 09:49:25,180|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\344DA\rR\275\031\256%<\255C(\0340\332\251Ou\r\357\323\3307\220\257\354Z\227\025\213Q"
> 2019-03-19 09:49:25,180|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:25,180|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\026\006\371\033n\220w\310\304\271G5qZ\177M9\310\330\024\335\035\006\"(\030/\322\t\301\225W"
> 2019-03-19 09:49:25,181|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "?\213SA\235\235\"\252-WU*\020\2616]MK\0261:_G\261p\372\3646\242\3135@"
> 2019-03-19 09:49:25,181|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "?\365=\262\020}\255\267B\257\271M\2555\206\363@\310x\255!\257\256\244\354\260\306\367\215z4\364"
> 2019-03-19 09:49:25,181|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:25,181|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:25,182|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:25,182|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|on datanode 1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c
> 2019-03-19 09:49:25,182|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:25,182|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2019-03-19 09:49:25,183|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> 2019-03-19 09:49:25,183|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:247)
> 2019-03-19 09:49:25,183|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:217)
> 2019-03-19 09:49:25,183|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:256)
> 2019-03-19 09:49:25,184|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:292)
> 2019-03-19 09:49:25,184|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.prepareRead(BlockInputStream.java:226)
> 2019-03-19 09:49:25,184|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:150)
> 2019-03-19 09:49:25,184|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.read(KeyInputStream.java:232)
> 2019-03-19 09:49:25,184|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:126)
> 2019-03-19 09:49:25,185|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47)
> 2019-03-19 09:49:25,185|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.io.InputStream.read(InputStream.java:101)
> 2019-03-19 09:49:25,185|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
> 2019-03-19 09:49:25,185|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:98)
> 2019-03-19 09:49:25,186|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:48)
> 2019-03-19 09:49:25,186|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.execute(CommandLine.java:919)
> 2019-03-19 09:49:25,186|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.access$700(CommandLine.java:104)
> 2019-03-19 09:49:25,186|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
> 2019-03-19 09:49:25,186|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
> 2019-03-19 09:49:25,186|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
> 2019-03-19 09:49:25,187|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
> 2019-03-19 09:49:25,187|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
> 2019-03-19 09:49:25,187|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
> 2019-03-19 09:49:25,187|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.execute(Shell.java:84)
> 2019-03-19 09:49:25,188|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
> 2019-03-19 09:49:25,188|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:95)
> 2019-03-19 09:49:25,188|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:25,188|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:526)
> 2019-03-19 09:49:25,188|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434)
> 2019-03-19 09:49:25,188|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:25,189|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:25,189|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:25,189|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678)
> 2019-03-19 09:49:25,190|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:25,190|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:25,190|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:25,190|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
> 2019-03-19 09:49:25,191|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> 2019-03-19 09:49:25,191|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> 2019-03-19 09:49:25,191|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> 2019-03-19 09:49:25,191|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> 2019-03-19 09:49:25,192|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> 2019-03-19 09:49:25,192|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2019-03-19 09:49:25,192|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> 2019-03-19 09:49:25,193|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2019-03-19 09:49:25,193|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2019-03-19 09:49:25,193|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.lang.Thread.run(Thread.java:748)
> 2019-03-19 09:49:25,193|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.27.89.128:9859
> 2019-03-19 09:49:25,193|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 2019-03-19 09:49:25,194|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> 2019-03-19 09:49:25,194|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
> 2019-03-19 09:49:25,194|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
> 2019-03-19 09:49:25,194|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632)
> 2019-03-19 09:49:25,195|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
> 2019-03-19 09:49:25,195|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
> 2019-03-19 09:49:25,195|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> 2019-03-19 09:49:25,195|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> 2019-03-19 09:49:25,196|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2019-03-19 09:49:25,196|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 1 more
> 2019-03-19 09:49:25,196|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: java.net.ConnectException: Connection refused
> 2019-03-19 09:49:25,196|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 11 more
> 2019-03-19 09:49:25,418|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|19/03/19 09:49:25 WARN scm.XceiverClientGrpc: Failed to execute command cmdType: ReadChunk
> 2019-03-19 09:49:25,418|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|traceID: "f5bb707b-b0ef-4c21-a6ea-910b72221cfd"
> 2019-03-19 09:49:25,418|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 2
> 2019-03-19 09:49:25,418|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|datanodeUuid: "1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c"
> 2019-03-19 09:49:25,419|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|readChunk {
> 2019-03-19 09:49:25,419|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockID {
> 2019-03-19 09:49:25,419|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 2
> 2019-03-19 09:49:25,419|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|localID: 101776681938518017
> 2019-03-19 09:49:25,419|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockCommitSequenceId: 43
> 2019-03-19 09:49:25,420|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:25,420|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkData {
> 2019-03-19 09:49:25,420|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkName: "f42c4a33a0b766f6a39981411a77be83_stream_fce2ae16-39ce-4f43-bf29-3db715ea2ced_chunk_6"
> 2019-03-19 09:49:25,420|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|offset: 0
> 2019-03-19 09:49:25,420|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|len: 16777216
> 2019-03-19 09:49:25,421|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksumData {
> 2019-03-19 09:49:25,421|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|type: SHA256
> 2019-03-19 09:49:25,421|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|bytesPerChecksum: 1048576
> 2019-03-19 09:49:25,421|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\020\261\363\207\220\304_@3C&\350i\274\225\034\0275\310\343xF\n\202&\252+\3361\2241T"
> 2019-03-19 09:49:25,421|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "}\314\243\235,\306\022\246\357\31289&\326\3030\303w\277\333@Y\032Z\366\244Q\3358.\027\334"
> 2019-03-19 09:49:25,422|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\347a\305\204\264\330\332\253\"\322\213DN6VUG\2253\274\022gLg\362\322\215I>\300\314\322"
> 2019-03-19 09:49:25,422|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\226\202\344\002\257\364\255\371\242F\314PF\fr\301\226P\037\302\362\207\302\021\376\254q\377\215\242:\303"
> 2019-03-19 09:49:25,422|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "k\242c\n\230c\320\235$w2\314\255\273\335\313\250\304lD\246\324\032\316\276\022\276%\341F\276\016"
> 2019-03-19 09:49:25,422|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\262|\'\353\213_\356{\341n\333\376\023\212^\000\255\205R\246\243\300\213\275\002\036* \177\356\332r"
> 2019-03-19 09:49:25,423|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\203\270\215\214\347$\341\004\363#\214\262\322\366;\336\336\253\027/\206\037E\247\027\303\'\260\2309\360\034"
> 2019-03-19 09:49:25,423|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: ":\023\213\236\364\210\v\'^?\322\227\232\321}\347$6\313o\256C6\331\341\"!\365\364\255\222\257"
> 2019-03-19 09:49:25,423|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "Z\226}NM\223G\370\362\216\365 [\002V\210\vD7_z.\266\327\033\021\366$\233\223Z\243"
> 2019-03-19 09:49:25,423|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\t\200\225\271\350\204L\221\343m\256\264\215[\343J\271f@\253\002\214V\240\002w\r\361k\020?^"
> 2019-03-19 09:49:25,423|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\245)-\005 \241\t\317\236\222\rt\"OJLM\355\353\352[\376NkrA\032\374\227\322\2471"
> 2019-03-19 09:49:25,424|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\316!+\3636#\315\262&\205\031\226\330\036\347\252\213S6w\371\337Bk>\301UGT\202{\363"
> 2019-03-19 09:49:25,424|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "0\341IU\353\3615\"f\334/\370\006~h\020F\a\347P\253\271\323\263e\202\270\257\220\237\313X"
> 2019-03-19 09:49:25,424|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\334Xz>(\2725@\320\214\233V\340J\242hO\344\264\217\260\"\305\200\203EgS\241\356\351/"
> 2019-03-19 09:49:25,424|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\204G\223\305\374v\233&\362\027\003~7\225\243\212D\270\211\2250Y$\321\0230Z\207\025\276\341|"
> 2019-03-19 09:49:25,424|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\020]\204k\336\323\311\227,\215h\237~\363F\236\r\230\201\323h\316\233\034\207\335\313f\024\005B\314"
> 2019-03-19 09:49:25,425|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:25,425|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:25,425|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:25,425|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|on datanode 1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c
> 2019-03-19 09:49:25,425|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:25,426|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2019-03-19 09:49:25,426|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> 2019-03-19 09:49:25,426|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:247)
> 2019-03-19 09:49:25,426|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:217)
> 2019-03-19 09:49:25,427|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:256)
> 2019-03-19 09:49:25,427|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:292)
> 2019-03-19 09:49:25,427|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.prepareRead(BlockInputStream.java:226)
> 2019-03-19 09:49:25,427|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:150)
> 2019-03-19 09:49:25,427|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.read(KeyInputStream.java:232)
> 2019-03-19 09:49:25,428|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:126)
> 2019-03-19 09:49:25,428|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47)
> 2019-03-19 09:49:25,428|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.io.InputStream.read(InputStream.java:101)
> 2019-03-19 09:49:25,428|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
> 2019-03-19 09:49:25,429|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:98)
> 2019-03-19 09:49:25,429|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:48)
> 2019-03-19 09:49:25,429|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.execute(CommandLine.java:919)
> 2019-03-19 09:49:25,429|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.access$700(CommandLine.java:104)
> 2019-03-19 09:49:25,429|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
> 2019-03-19 09:49:25,430|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
> 2019-03-19 09:49:25,430|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
> 2019-03-19 09:49:25,430|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
> 2019-03-19 09:49:25,430|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
> 2019-03-19 09:49:25,430|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
> 2019-03-19 09:49:25,431|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.execute(Shell.java:84)
> 2019-03-19 09:49:25,431|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
> 2019-03-19 09:49:25,431|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:95)
> 2019-03-19 09:49:25,432|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:25,432|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:526)
> 2019-03-19 09:49:25,432|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434)
> 2019-03-19 09:49:25,432|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:25,433|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:25,433|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:25,433|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678)
> 2019-03-19 09:49:25,433|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:25,433|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:25,434|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:25,434|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
> 2019-03-19 09:49:25,434|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> 2019-03-19 09:49:25,434|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> 2019-03-19 09:49:25,434|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> 2019-03-19 09:49:25,435|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> 2019-03-19 09:49:25,435|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> 2019-03-19 09:49:25,435|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2019-03-19 09:49:25,435|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> 2019-03-19 09:49:25,435|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2019-03-19 09:49:25,436|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2019-03-19 09:49:25,436|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.lang.Thread.run(Thread.java:748)
> 2019-03-19 09:49:25,436|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.27.89.128:9859
> 2019-03-19 09:49:25,436|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 2019-03-19 09:49:25,436|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> 2019-03-19 09:49:25,437|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
> 2019-03-19 09:49:25,437|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
> 2019-03-19 09:49:25,437|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632)
> 2019-03-19 09:49:25,437|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
> 2019-03-19 09:49:25,438|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
> 2019-03-19 09:49:25,438|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> 2019-03-19 09:49:25,438|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> 2019-03-19 09:49:25,438|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2019-03-19 09:49:25,438|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 1 more
> 2019-03-19 09:49:25,439|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: java.net.ConnectException: Connection refused
> 2019-03-19 09:49:25,439|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 11 more
> 2019-03-19 09:49:25,805|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|19/03/19 09:49:25 WARN scm.XceiverClientGrpc: Failed to execute command cmdType: ReadChunk
> 2019-03-19 09:49:25,806|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|traceID: "f5bb707b-b0ef-4c21-a6ea-910b72221cfd"
> 2019-03-19 09:49:25,806|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 2
> 2019-03-19 09:49:25,806|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|datanodeUuid: "1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c"
> 2019-03-19 09:49:25,806|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|readChunk {
> 2019-03-19 09:49:25,806|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockID {
> 2019-03-19 09:49:25,807|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 2
> 2019-03-19 09:49:25,807|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|localID: 101776681938518017
> 2019-03-19 09:49:25,807|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockCommitSequenceId: 43
> 2019-03-19 09:49:25,808|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:25,808|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkData {
> 2019-03-19 09:49:25,808|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkName: "f42c4a33a0b766f6a39981411a77be83_stream_fce2ae16-39ce-4f43-bf29-3db715ea2ced_chunk_7"
> 2019-03-19 09:49:25,808|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|offset: 0
> 2019-03-19 09:49:25,808|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|len: 16777216
> 2019-03-19 09:49:25,809|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksumData {
> 2019-03-19 09:49:25,809|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|type: SHA256
> 2019-03-19 09:49:25,809|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|bytesPerChecksum: 1048576
> 2019-03-19 09:49:25,809|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\210\257\216\346Y\336\260\326m\r\267-\317\205.\214\231\300\317zzjo\355\347\246,\370\370)J\205"
> 2019-03-19 09:49:25,810|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\245A\362yn\343d\273\313\370\355m\341\237\306SS-ch\222\b\366^\035\230n\233\224$\243L"
> 2019-03-19 09:49:25,810|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "!3\016\216\302\0051\350@&\200\251\343|\027\357\252\233\326\226\270\3010Y\004\027C\310\337a\352\231"
> 2019-03-19 09:49:25,810|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\251\304\005\301W\266f\225?\276\344W\f\016\235\344\213\b\300\276\200_\310!\375M\305\307\021\337cB"
> 2019-03-19 09:49:25,810|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\242*K\253\320V?\355\251\033\002l\226\002\030\351\256\243\337\377\242x(\f\266\375\"\rb\313\270G"
> 2019-03-19 09:49:25,811|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\365q1\037\312\a\322\262`\211\252\204\342U#\341\271[\023;\323F\270_\222p*3\371\235\f\247"
> 2019-03-19 09:49:25,811|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\321\244\321\256\036\314.\322\327\366\323\217\034!\247\264K\004D\250\214\213\202\257\370k\317v\225a\266K"
> 2019-03-19 09:49:25,811|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\227m\311\270\214`\345\352|\254\212\336B:\244ix\212\032bFr\273\253\313\231\307\237\312\263\271\270"
> 2019-03-19 09:49:25,811|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\376pi\033*k\261m\366r3jy\255\203\271\376\217\205\346\225{\203\207\027\265\t\225:2\030A"
> 2019-03-19 09:49:25,812|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "k\233\277\020\275K\221x]i\237ZIoGt\240RSv\201\003\227D\357\020\005\213\264\236\006\030"
> 2019-03-19 09:49:25,812|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "H\325\271\222n<\202\250\020sR\241{HD\320\377\f\330\004\320\231N\314\320j\204\365ah\227\204"
> 2019-03-19 09:49:25,812|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\344\263\246s\2554\237q\356\326\210\347\314\035\0022\020\005\033l\'\215(\225\376=\025\301tmlW"
> 2019-03-19 09:49:25,812|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "X\323\206\326P\341\257\216,o\324\354@:Fz\361\222\217\350\006\265 \021\030 \240\355\'\3414\362"
> 2019-03-19 09:49:25,813|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "~\337\270\261(\247\365\225\306\310\020\240\271X\375\035\364ZO=(\233G\031K\032T\335\003\356B\374"
> 2019-03-19 09:49:25,813|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "(\225\307\202\023%\201\365]a\336\241\371\276\025\376\260\310~\315\202\306\355\240v<\272%N\271\230:"
> 2019-03-19 09:49:25,813|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\204/\313\366\242\230\201\033\343\353\326\244\2317 7\367\264\027\035\225\314\000ld\342\347\364\n/y\303"
> 2019-03-19 09:49:25,814|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:25,814|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:25,814|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:25,814|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|on datanode 1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c
> 2019-03-19 09:49:25,815|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:25,815|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2019-03-19 09:49:25,815|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> 2019-03-19 09:49:25,815|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:247)
> 2019-03-19 09:49:25,815|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:217)
> 2019-03-19 09:49:25,816|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:256)
> 2019-03-19 09:49:25,816|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:292)
> 2019-03-19 09:49:25,816|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.prepareRead(BlockInputStream.java:226)
> 2019-03-19 09:49:25,816|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:150)
> 2019-03-19 09:49:25,817|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.read(KeyInputStream.java:232)
> 2019-03-19 09:49:25,817|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:126)
> 2019-03-19 09:49:25,817|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47)
> 2019-03-19 09:49:25,817|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.io.InputStream.read(InputStream.java:101)
> 2019-03-19 09:49:25,817|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
> 2019-03-19 09:49:25,818|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:98)
> 2019-03-19 09:49:25,818|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:48)
> 2019-03-19 09:49:25,818|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.execute(CommandLine.java:919)
> 2019-03-19 09:49:25,818|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.access$700(CommandLine.java:104)
> 2019-03-19 09:49:25,818|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
> 2019-03-19 09:49:25,819|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
> 2019-03-19 09:49:25,819|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
> 2019-03-19 09:49:25,819|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
> 2019-03-19 09:49:25,819|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
> 2019-03-19 09:49:25,819|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
> 2019-03-19 09:49:25,819|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.execute(Shell.java:84)
> 2019-03-19 09:49:25,820|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
> 2019-03-19 09:49:25,820|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:95)
> 2019-03-19 09:49:25,820|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:25,820|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:526)
> 2019-03-19 09:49:25,820|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434)
> 2019-03-19 09:49:25,821|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:25,821|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:25,821|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:25,821|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678)
> 2019-03-19 09:49:25,822|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:25,822|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:25,822|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:25,822|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
> 2019-03-19 09:49:25,823|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> 2019-03-19 09:49:25,823|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> 2019-03-19 09:49:25,823|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> 2019-03-19 09:49:25,823|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> 2019-03-19 09:49:25,824|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> 2019-03-19 09:49:25,824|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2019-03-19 09:49:25,824|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> 2019-03-19 09:49:25,824|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2019-03-19 09:49:25,824|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2019-03-19 09:49:25,825|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.lang.Thread.run(Thread.java:748)
> 2019-03-19 09:49:25,825|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.27.89.128:9859
> 2019-03-19 09:49:25,825|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 2019-03-19 09:49:25,825|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> 2019-03-19 09:49:25,825|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
> 2019-03-19 09:49:25,826|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
> 2019-03-19 09:49:25,826|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632)
> 2019-03-19 09:49:25,826|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
> 2019-03-19 09:49:25,826|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
> 2019-03-19 09:49:25,827|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> 2019-03-19 09:49:25,827|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> 2019-03-19 09:49:25,827|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2019-03-19 09:49:25,827|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 1 more
> 2019-03-19 09:49:25,827|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: java.net.ConnectException: Connection refused
> 2019-03-19 09:49:25,828|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 11 more
> 2019-03-19 09:49:25,959|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|19/03/19 09:49:25 WARN scm.XceiverClientGrpc: Failed to execute command cmdType: ReadChunk
> 2019-03-19 09:49:25,959|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|traceID: "f5bb707b-b0ef-4c21-a6ea-910b72221cfd"
> 2019-03-19 09:49:25,960|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 2
> 2019-03-19 09:49:25,960|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|datanodeUuid: "1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c"
> 2019-03-19 09:49:25,960|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|readChunk {
> 2019-03-19 09:49:25,960|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockID {
> 2019-03-19 09:49:25,961|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|containerID: 2
> 2019-03-19 09:49:25,961|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|localID: 101776681938518017
> 2019-03-19 09:49:25,961|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|blockCommitSequenceId: 43
> 2019-03-19 09:49:25,962|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:25,962|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkData {
> 2019-03-19 09:49:25,962|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|chunkName: "f42c4a33a0b766f6a39981411a77be83_stream_fce2ae16-39ce-4f43-bf29-3db715ea2ced_chunk_8"
> 2019-03-19 09:49:25,962|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|offset: 0
> 2019-03-19 09:49:25,963|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|len: 15728640
> 2019-03-19 09:49:25,963|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksumData {
> 2019-03-19 09:49:25,963|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|type: SHA256
> 2019-03-19 09:49:25,963|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|bytesPerChecksum: 1048576
> 2019-03-19 09:49:25,963|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\221\371\346\003r\367\203\r\271o\243q\313dw}_\363\307\372\256^\362\361z\260\212V\206!\264\005"
> 2019-03-19 09:49:25,964|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "G\236\205@AF\357\006,V>>\262Hd\365\256loi\214\232\20260\316@~\376\360\222{"
> 2019-03-19 09:49:25,964|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\245\272\2037\266va\026z\342\vX|^\210\302\223\250|\243M\343\307-B!\313\366\'[\277]"
> 2019-03-19 09:49:25,964|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "c\243w\203/\310\036\210\364\034\262\353\211s\211W\251\303\362\231\230\336\a\362\254\307\3303\301\311\207%"
> 2019-03-19 09:49:25,965|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "M]\223-\277\333O\326\352\331\221{\311\221Y\317\314*\347\312\236\254\"\253H\\\016\207\342\312U\213"
> 2019-03-19 09:49:25,965|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "ag}\253\267@\377{\275a\277r\312\226\243\246\222\177c\337r\256\022\204\273@c\f0\367i\214"
> 2019-03-19 09:49:25,965|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\035\203SE\034\224L\253\353J`\'\b\241 FsY\273]\246\353\261\233=A\375\302\261\261\254\335"
> 2019-03-19 09:49:25,965|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "5@\221\005o\022\315\352\334\001\244\353_\003M\250h\364\361QxV\037\300\2137\305\205\335\220|\350"
> 2019-03-19 09:49:25,966|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "5\366\032\023\236\020\271Z2`q\345\n\316R4\316\306\2534\373\366\241N\223\222\341\302\330\362\260~"
> 2019-03-19 09:49:25,966|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\244_n\364\262[\255,\016\251{l\231N\365i>\244dg\250q\021C\300\321\304\355|\211\327\273"
> 2019-03-19 09:49:25,966|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "!\000\020\006A\372\3265\026h\363\373\375\224\261\264\200\241X\315vOE{a\214OBW\255_m"
> 2019-03-19 09:49:25,967|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "(y{\366 \327\215\017\000\367Ib\034UJ\347\312\037\202\203\005\352moA\2166\023;\243\316J"
> 2019-03-19 09:49:25,967|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\217\210\021(\203\t\2046\336\323\323\315\335\a\r\n\271\334\337\357\347c\263-7r\232\033T\317\271\344"
> 2019-03-19 09:49:25,967|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: "\257\242\031\204\340\235,[\274\323\216\260\367\277\333\030*X\245)<%\241\332\344\274LX\005\302\275A"
> 2019-03-19 09:49:25,968|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|checksums: ";\n\222\316\275C\021J}\274p\313X#\266\024M\347nkuW5\023r\242\027\205\370\261\305\300"
> 2019-03-19 09:49:25,968|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:25,968|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:25,968|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|}
> 2019-03-19 09:49:25,968|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|on datanode 1b0f65e6-2ac8-4c5f-b552-e96f3a262f8c
> 2019-03-19 09:49:25,969|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:25,969|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2019-03-19 09:49:25,969|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> 2019-03-19 09:49:25,970|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:247)
> 2019-03-19 09:49:25,970|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:217)
> 2019-03-19 09:49:25,970|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.readChunk(ContainerProtocolCalls.java:256)
> 2019-03-19 09:49:25,971|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readChunkFromContainer(BlockInputStream.java:292)
> 2019-03-19 09:49:25,971|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.prepareRead(BlockInputStream.java:226)
> 2019-03-19 09:49:25,971|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:150)
> 2019-03-19 09:49:25,971|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream$ChunkInputStreamEntry.read(KeyInputStream.java:232)
> 2019-03-19 09:49:25,972|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:126)
> 2019-03-19 09:49:25,972|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47)
> 2019-03-19 09:49:25,972|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.io.InputStream.read(InputStream.java:101)
> 2019-03-19 09:49:25,973|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:100)
> 2019-03-19 09:49:25,973|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:98)
> 2019-03-19 09:49:25,973|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.keys.GetKeyHandler.call(GetKeyHandler.java:48)
> 2019-03-19 09:49:25,973|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.execute(CommandLine.java:919)
> 2019-03-19 09:49:25,973|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.access$700(CommandLine.java:104)
> 2019-03-19 09:49:25,974|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1083)
> 2019-03-19 09:49:25,974|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$RunLast.handle(CommandLine.java:1051)
> 2019-03-19 09:49:25,974|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:959)
> 2019-03-19 09:49:25,974|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandlers(CommandLine.java:1242)
> 2019-03-19 09:49:25,974|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at picocli.CommandLine.parseWithHandler(CommandLine.java:1181)
> 2019-03-19 09:49:25,975|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.execute(GenericCli.java:61)
> 2019-03-19 09:49:25,975|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.execute(Shell.java:84)
> 2019-03-19 09:49:25,975|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.hdds.cli.GenericCli.run(GenericCli.java:52)
> 2019-03-19 09:49:25,975|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.hadoop.ozone.web.ozShell.Shell.main(Shell.java:95)
> 2019-03-19 09:49:25,976|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2019-03-19 09:49:25,976|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:526)
> 2019-03-19 09:49:25,976|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434)
> 2019-03-19 09:49:25,977|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:25,978|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:25,978|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:25,980|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678)
> 2019-03-19 09:49:25,981|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2019-03-19 09:49:25,981|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2019-03-19 09:49:25,982|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> 2019-03-19 09:49:25,983|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
> 2019-03-19 09:49:25,984|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> 2019-03-19 09:49:25,984|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> 2019-03-19 09:49:25,984|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> 2019-03-19 09:49:25,986|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> 2019-03-19 09:49:25,986|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> 2019-03-19 09:49:25,987|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2019-03-19 09:49:25,988|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> 2019-03-19 09:49:25,988|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2019-03-19 09:49:25,989|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2019-03-19 09:49:25,989|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at java.lang.Thread.run(Thread.java:748)
> 2019-03-19 09:49:25,990|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.27.89.128:9859
> 2019-03-19 09:49:25,990|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> 2019-03-19 09:49:25,990|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> 2019-03-19 09:49:25,990|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
> 2019-03-19 09:49:25,991|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
> 2019-03-19 09:49:25,991|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632)
> 2019-03-19 09:49:25,991|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
> 2019-03-19 09:49:25,991|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
> 2019-03-19 09:49:25,991|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
> 2019-03-19 09:49:25,992|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> 2019-03-19 09:49:25,992|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2019-03-19 09:49:25,992|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 1 more
> 2019-03-19 09:49:25,992|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Caused by: java.net.ConnectException: Connection refused
> 2019-03-19 09:49:25,993|INFO|MainThread|machine.py:180 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|... 11 more
> 2019-03-19 09:49:26,621|INFO|MainThread|machine.py:207 - run()||GUID=4bc13d85-dd87-4806-b3cc-7cc12fce8fdc|Exit Code: 0
> 2019-03-19 09:49:26,732|INFO|MainThread|machine.py:165 - run()||GUID=bf8a74f6-73bb-47e4-b572-47bbf824c543|RUNNING: ssh -l root -i /tmp/hw-qe-keypair.pem -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ctr-e139-1542663976389-88627-01-000002.hwx.site "sudo su - -c \"md5sum /tmp/OzonePipelineTest/WriteOzoneFileGet_1552988856\" root"
> 2019-03-19 09:49:30,213|INFO|MainThread|machine.py:180 - run()||GUID=bf8a74f6-73bb-47e4-b572-47bbf824c543|665facf9b10136e2b228ed64aaee80f2 /tmp/OzonePipelineTest/WriteOzoneFileGet_1552988856
> 2019-03-19 09:49:30,214|INFO|MainThread|machine.py:207 - run()||GUID=bf8a74f6-73bb-47e4-b572-47bbf824c543|Exit Code: 0
> 2019-03-19 09:49:30,214|INFO|MainThread|machine.py:642 - find_checksum()|Checksum of /tmp/OzonePipelineTest/WriteOzoneFileGet_1552988856 is : 665facf9b10136e2b228ed64aaee80f2
> 2019-03-19 09:49:30,216|INFO|MainThread|machine.py:165 - run()||GUID=c6f6b76e-4b17-4bdd-a595-a038a8e3262d|RUNNING: ssh -l root -i /tmp/hw-qe-keypair.pem -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ctr-e139-1542663976389-88627-01-000002.hwx.site "sudo su - -c \"ls /tmp/OzonePipelineTest/WriteOzoneFileGet_1552988856\" root"
> 2019-03-19 09:49:33,129|INFO|MainThread|machine.py:180 - run()||GUID=c6f6b76e-4b17-4bdd-a595-a038a8e3262d|/tmp/OzonePipelineTest/WriteOzoneFileGet_1552988856
> 2019-03-19 09:49:33,132|INFO|MainThread|machine.py:207 - run()||GUID=c6f6b76e-4b17-4bdd-a595-a038a8e3262d|Exit Code: 0
> 2019-03-19 09:49:33,133|INFO|MainThread|ambari.py:408 - getAdminPassword()|NoSectionError Exception raise. setting hdc = no by default.
> 2019-03-19 09:49:34,265|INFO|MainThread|ambari.py:408 - getAdminPassword()|NoSectionError Exception raise. setting hdc = no by default.
> 2019-03-19 09:49:34,265|INFO|MainThread|ambari.py:550 - startComponent()|Initiating start of DATANODE at host ctr-e139-1542663976389-88627-01-000002.hwx.site, request url is: https://172.27.89.128/ambari/api/v1/clusters/ozone-190317-133635-l3p/hosts/ctr-e139-1542663976389-88627-01-000002.hwx.site/host_components/DATANODE
> 2019-03-19 09:49:35,678|INFO|MainThread|ambari.py:561 - startComponent()|Completed start of DATANODE at host ctr-e139-1542663976389-88627-01-000002.hwx.site
> 2019-03-19 09:49:35,679|INFO|MainThread|conftest.py:345 - pytest_runtest_makereport()|call: <CallInfo when='call' result: []>
> 2019-03-19 09:49:35,680|INFO|MainThread|conftest.py:346 - pytest_runtest_makereport()|item: <Function 'test_ShutdownOneDatanodeWhileWrite'>
> 2019-03-19 09:49:35,681|INFO|MainThread|conftest.py:337 - pytest_report_teststatus()|TEST "test_ShutdownOneDatanodeWhileWrite" PASSED in 119.31 seconds
> .2019-03-19 09:49:35,682|INFO|MainThread|ozone.py:109 - listKey()|Running list key in test-volume1-1552988834/test-bucket1-1552988834
> 2019-03-19 09:49:35,683|INFO|MainThread|machine.py:165 - run()||GUID=6da81118-ca4f-457c-a80b-c0a07a89e341|RUNNING: ssh -l root -i /tmp/hw-qe-keypair.pem -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ctr-e139-1542663976389-88627-01-000002.hwx.site "sudo su - -c \"/usr/hdp/current/hadoop-ozone/bin/ozone sh key list /test-volume1-1552988834/test-bucket1-1552988834/\" root"
> 2019-03-19 09:49:39,338|INFO|MainThread|machine.py:180 - run()||GUID=6da81118-ca4f-457c-a80b-c0a07a89e341|SLF4J: Class path contains multiple SLF4J bindings.
> 2019-03-19 09:49:39,339|INFO|MainThread|machine.py:180 - run()||GUID=6da81118-ca4f-457c-a80b-c0a07a89e341|SLF4J: Found binding in [jar:file:/usr/hdp/3.0.100.0-348/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 2019-03-19 09:49:39,340|INFO|MainThread|machine.py:180 - run()||GUID=6da81118-ca4f-457c-a80b-c0a07a89e341|SLF4J: Found binding in [jar:file:/usr/hdp/3.0.100.0-348/hadoop-ozone/share/ozone/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 2019-03-19 09:49:39,340|INFO|MainThread|machine.py:180 - run()||GUID=6da81118-ca4f-457c-a80b-c0a07a89e341|SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> 2019-03-19 09:49:39,366|INFO|MainThread|machine.py:180 - run()||GUID=6da81118-ca4f-457c-a80b-c0a07a89e341|SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 2019-03-19 09:49:41,357|INFO|MainThread|machine.py:180 - run()||GUID=6da81118-ca4f-457c-a80b-c0a07a89e341|[ {
> 2019-03-19 09:49:41,359|INFO|MainThread|machine.py:180 - run()||GUID=6da81118-ca4f-457c-a80b-c0a07a89e341|"version" : 0,
> 2019-03-19 09:49:41,359|INFO|MainThread|machine.py:180 - run()||GUID=6da81118-ca4f-457c-a80b-c0a07a89e341|"md5hash" : null,
> 2019-03-19 09:49:41,360|INFO|MainThread|machine.py:180 - run()||GUID=6da81118-ca4f-457c-a80b-c0a07a89e341|"createdOn" : "Tue, 19 Mar 2019 09:48:05 GMT",
> 2019-03-19 09:49:41,360|INFO|MainThread|machine.py:180 - run()||GUID=6da81118-ca4f-457c-a80b-c0a07a89e341|"modifiedOn" : "Tue, 19 Mar 2019 09:49:13 GMT",
> 2019-03-19 09:49:41,360|INFO|MainThread|machine.py:180 - run()||GUID=6da81118-ca4f-457c-a80b-c0a07a89e341|"size" : 267386880,
> 2019-03-19 09:49:41,361|INFO|MainThread|machine.py:180 - run()||GUID=6da81118-ca4f-457c-a80b-c0a07a89e341|"keyName" : "WriteOzoneFile_1552988856",
> 2019-03-19 09:49:41,361|INFO|MainThread|machine.py:180 - run()||GUID=6da81118-ca4f-457c-a80b-c0a07a89e341|"type" : null
> 2019-03-19 09:49:41,361|INFO|MainThread|machine.py:180 - run()||GUID=6da81118-ca4f-457c-a80b-c0a07a89e341|} ]
> 2019-03-19 09:49:41,385|INFO|MainThread|machine.py:207 - run()||GUID=6da81118-ca4f-457c-a80b-c0a07a89e341|Exit Code: 0
> 2019-03-19 09:49:41,386|INFO|MainThread|ozone.py:112 - listKey()|output=[[ {
> "version" : 0,
> "md5hash" : null,
> "createdOn" : "Tue, 19 Mar 2019 09:48:05 GMT",
> "modifiedOn" : "Tue, 19 Mar 2019 09:49:13 GMT",
> "size" : 267386880,
> "keyName" : "WriteOzoneFile_1552988856",
> "type" : null
> } ]]
> 2019-03-19 09:49:41,386|INFO|MainThread|ozone.py:123 - deleteKey()|Running delete key WriteOzoneFile_1552988856 in test-volume1-1552988834/test-bucket1-1552988834
> 2019-03-19 09:49:41,388|INFO|MainThread|machine.py:165 - run()||GUID=11f71a4d-2ca0-4f04-ba4d-318123b72d78|RUNNING: ssh -l root -i /tmp/hw-qe-keypair.pem -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ctr-e139-1542663976389-88627-01-000002.hwx.site "sudo su - -c \"/usr/hdp/current/hadoop-ozone/bin/ozone sh key delete /test-volume1-1552988834/test-bucket1-1552988834/WriteOzoneFile_1552988856\" root"
> 2019-03-19 09:49:44,943|INFO|MainThread|machine.py:180 - run()||GUID=11f71a4d-2ca0-4f04-ba4d-318123b72d78|SLF4J: Class path contains multiple SLF4J bindings.
> 2019-03-19 09:49:44,949|INFO|MainThread|machine.py:180 - run()||GUID=11f71a4d-2ca0-4f04-ba4d-318123b72d78|SLF4J: Found binding in [jar:file:/usr/hdp/3.0.100.0-348/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 2019-03-19 09:49:44,949|INFO|MainThread|machine.py:180 - run()||GUID=11f71a4d-2ca0-4f04-ba4d-318123b72d78|SLF4J: Found binding in [jar:file:/usr/hdp/3.0.100.0-348/hadoop-ozone/share/ozone/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 2019-03-19 09:49:44,950|INFO|MainThread|machine.py:180 - run()||GUID=11f71a4d-2ca0-4f04-ba4d-318123b72d78|SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> 2019-03-19 09:49:45,015|INFO|MainThread|machine.py:180 - run()||GUID=11f71a4d-2ca0-4f04-ba4d-318123b72d78|SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 2019-03-19 09:49:46,793|INFO|MainThread|machine.py:207 - run()||GUID=11f71a4d-2ca0-4f04-ba4d-318123b72d78|Exit Code: 0
> 2019-03-19 09:49:46,795|INFO|MainThread|ozone.py:70 - deleteBucket()|Running delete bucket of test-volume1-1552988834/test-bucket1-1552988834
> 2019-03-19 09:49:46,796|INFO|MainThread|machine.py:165 - run()||GUID=5477cd27-e759-4a0d-8ac1-3cb2e8c28359|RUNNING: ssh -l root -i /tmp/hw-qe-keypair.pem -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ctr-e139-1542663976389-88627-01-000002.hwx.site "sudo su - -c \"/usr/hdp/current/hadoop-ozone/bin/ozone sh bucket delete /test-volume1-1552988834/test-bucket1-1552988834\" root"
> 2019-03-19 09:49:50,237|INFO|MainThread|machine.py:180 - run()||GUID=5477cd27-e759-4a0d-8ac1-3cb2e8c28359|SLF4J: Class path contains multiple SLF4J bindings.
> 2019-03-19 09:49:50,237|INFO|MainThread|machine.py:180 - run()||GUID=5477cd27-e759-4a0d-8ac1-3cb2e8c28359|SLF4J: Found binding in [jar:file:/usr/hdp/3.0.100.0-348/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 2019-03-19 09:49:50,238|INFO|MainThread|machine.py:180 - run()||GUID=5477cd27-e759-4a0d-8ac1-3cb2e8c28359|SLF4J: Found binding in [jar:file:/usr/hdp/3.0.100.0-348/hadoop-ozone/share/ozone/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> 2019-03-19 09:49:50,238|INFO|MainThread|machine.py:180 - run()||GUID=5477cd27-e759-4a0d-8ac1-3cb2e8c28359|SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> 2019-03-19 09:49:50,278|INFO|MainThread|machine.py:180 - run()||GUID=5477cd27-e759-4a0d-8ac1-3cb2e8c28359|SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 2019-03-19 09:49:52,231|INFO|MainThread|machine.py:207 - run()||GUID=5477cd27-e759-4a0d-8ac1-3cb2e8c28359|Exit Code: 0{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org