You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2021/04/13 08:48:00 UTC
[jira] [Updated] (HDDS-4986) Read failure because of unhealthy
container
[ https://issues.apache.org/jira/browse/HDDS-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated HDDS-4986:
---------------------------------
Labels: pull-request-available (was: )
> Read failure because of unhealthy container
> -------------------------------------------
>
> Key: HDDS-4986
> URL: https://issues.apache.org/jira/browse/HDDS-4986
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Sammi Chen
> Assignee: Sammi Chen
> Priority: Critical
> Labels: pull-request-available
>
> Failed to read user data because of container which contains data is of unhealthy state.
> aws CLI response,
> get file 10.121.81.124/0ea3094efb41471c861f25d9c429d9f6.jfr
> An error occurred (500) when calling the GetObject operation (reached max retries: 4): Internal Server Error
> s3g LOG,
> Caused by: org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: The container(1839) replica is unhealthy.
> at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.validateContainerResponse(ContainerProtocolCalls.java:530)
> at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.lambda$getValidatorList$0(ContainerProtocolCalls.java:537)
> at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:374)
> at org.apache.hadoop.hdds.scm.XceiverClientGrpc.lambda$sendCommandWithTraceIDAndRetry$0(XceiverClientGrpc.java:311)
> at org.apache.hadoop.hdds.tracing.TracingUtil.executeInSpan(TracingUtil.java:174)
> at org.apache.hadoop.hdds.tracing.TracingUtil.executeInNewSpan(TracingUtil.java:148)
> at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:305)
> at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:286)
> at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:106)
> at org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:213)
> at org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:142)
> at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:276)
> at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:199)
> at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:49)
> at java.io.InputStream.read(InputStream.java:101)
> at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2146)
> at org.apache.commons.io.IOUtils.copy(IOUtils.java:2102)
> at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2123)
> at org.apache.commons.io.IOUtils.copy(IOUtils.java:2078)
> at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.lambda$get$0(ObjectEndpoint.java:278)
> at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:79)
> at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:61)
> at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:266)
> at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:251)
> at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
> at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:109)
> at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:163)
> at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:85)
> ... 62 more
> For such containers, it is in CLOSING state,
> bin/ozone admin container info 1839
> Container id: 1839
> Pipeline id: a9642446-33c9-42ae-9161-7c3a570cea93
> Container State: CLOSING
> Datanodes: [32e8d855-b702-438d-b829-ac43dc567afc/tdw-9-180-19-146,
> b85962f2-6647-463b-9944-3c9b24e4e313/tdw-9-180-19-148,
> 07b24dc5-1d5e-4f6e-ad15-e3d89ea06c54/tdw-9-180-19-145]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org