You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Stephen O'Donnell (Jira)" <ji...@apache.org> on 2022/10/11 09:49:00 UTC
[jira] [Resolved] (HDDS-7304) EC: EC Decode can fail when byteBuffer from elastic pool is larger than chunksize
[ https://issues.apache.org/jira/browse/HDDS-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stephen O'Donnell resolved HDDS-7304.
-------------------------------------
Fix Version/s: 1.3.0
Resolution: Fixed
> EC: EC Decode can fail when byteBuffer from elastic pool is larger than chunksize
> ---------------------------------------------------------------------------------
>
> Key: HDDS-7304
> URL: https://issues.apache.org/jira/browse/HDDS-7304
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: EC Client
> Reporter: Stephen O'Donnell
> Assignee: Stephen O'Donnell
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.3.0
>
>
> The EC client establishes an ElasticByteBufferPool, so that byte buffers can be assigned / freed and reused without allocating and freeing memory over and over. The buffer returned by the elastic buffer can be larger than the size requested, and if that is the case we can get an error on EC decode, such as:
> {code}
> java.lang.IllegalArgumentException: Invalid buffer [3], not of length 1048576
> at org.apache.ozone.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:106)
> at org.apache.ozone.erasurecode.rawcoder.ByteBufferDecodingState.<init>(ByteBufferDecodingState.java:44)
> at org.apache.ozone.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:84)
> at org.apache.hadoop.ozone.client.io.ECBlockReconstructedStripeInputStream.decodeStripe(ECBlockReconstructedStripeInputStream.java:649)
> at org.apache.hadoop.ozone.client.io.ECBlockReconstructedStripeInputStream.read(ECBlockReconstructedStripeInputStream.java:388)
> at org.apache.hadoop.ozone.client.io.TestECBlockReconstructedStripeInputStream.testErrorReadingBlockContinuesReading(TestECBlockReconstructedStripeInputStream.java:622)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
> at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> ...
> {code}
> The solution is to set the limit on the buffer to the appropriate size before attempting to use it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org