You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Mukul Kumar Singh (Jira)" <ji...@apache.org> on 2020/03/31 04:52:00 UTC
[jira] [Assigned] (HDDS-3293) read operation failing when two
container replicas are corrupted
[ https://issues.apache.org/jira/browse/HDDS-3293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukul Kumar Singh reassigned HDDS-3293:
---------------------------------------
Assignee: Shashikant Banerjee
> read operation failing when two container replicas are corrupted
> ----------------------------------------------------------------
>
> Key: HDDS-3293
> URL: https://issues.apache.org/jira/browse/HDDS-3293
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
> Reporter: Nilotpal Nandi
> Assignee: Shashikant Banerjee
> Priority: Major
> Labels: fault_injection
>
> steps taken :
> 1) Mounted noise injection FUSE on all datanodes.
> 2) Write a key ( multi blocks)
> 3) Select one of the container ids , inject error on 2 container replicas for that container id.
> 4) Run GET key operation.
> GET key operation fails intermittenly.
> Error seen :
> -------------
>
> {noformat}
> 20/03/27 18:30:40 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-xceiverclientmetrics.properties,hadoop-metrics2.properties
> E 20/03/27 18:30:40 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
> E 20/03/27 18:30:40 INFO impl.MetricsSystemImpl: XceiverClientMetrics metrics system started
> E 20/03/27 18:31:12 ERROR scm.XceiverClientGrpc: Failed to execute command cmdType: ReadChunk
> E traceID: "f80a51eaec481a1c:cbb8e92869015a53:f80a51eaec481a1c:0"
> E containerID: 67
> E datanodeUuid: "96101390-2446-40e6-a54e-36e170497e57"
> E readChunk {
> E blockID {
> E containerID: 67
> E localID: 103896435892617248
> E blockCommitSequenceId: 1010
> E }
> E chunkData {
> E chunkName: "103896435892617248_chunk_28"
> E offset: 113246208
> E len: 4194304
> E checksumData {
> E type: CRC32
> E bytesPerChecksum: 1048576
> E checksums: "\034\376\313\031"
> E checksums: ";U\225\037"
> E checksums: "\327m\332."
> E checksums: "|\307\004E"
> E }
> E }
> E }
> E on the pipeline Pipeline[ Id: bce6316c-9690-452b-80e3-0f3590533444, Nodes: 96101390-2446-40e6-a54e-36e170497e57{ip: 172.27.111.129, host: quasar-olrywk-3.quasar-olrywk.root.hwx.site, networkLocation: /default-rack, certSerialId: null}3e85204d-2399-43b5-952a-55b837eb4c1d{ip: 172.27.100.0, host: quasar-olrywk-1.quasar-olrywk.root.hwx.site, networkLocation: /default-rack, certSerialId: null}5af0340a-6fee-4ce8-9f68-37fa35566a5a{ip: 172.27.73.0, host: quasar-olrywk-9.quasar-olrywk.root.hwx.site, networkLocation: /default-rack, certSerialId: null}, Type:STAND_ALONE, Factor:THREE, State:OPEN, leaderId:96101390-2446-40e6-a54e-36e170497e57, CreationTimestamp2020-03-27T03:36:51.880Z].
> E Unexpected OzoneException: java.io.IOException: java.util.concurrent.ExecutionException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 84603913ns. [remote_addr=/172.27.73.0:9859]]{noformat}
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org