You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/10/10 16:43:54 UTC
[GitHub] [ozone] sodonnel opened a new pull request, #3817: HDDS-7304. EC: EC Decode can fail when byteBuffer from elastic pool is larger than chunksize
sodonnel opened a new pull request, #3817:
URL: https://github.com/apache/ozone/pull/3817
## What changes were proposed in this pull request?
he EC client establishes an ElasticByteBufferPool, so that byte buffers can be assigned / freed and reused without allocating and freeing memory over and over. The buffer returned by the elastic buffer can be larger than the size requested, and if that is the case we can get an error on EC decode, such as:
```
java.lang.IllegalArgumentException: Invalid buffer [3], not of length 1048576
at org.apache.ozone.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:106)
at org.apache.ozone.erasurecode.rawcoder.ByteBufferDecodingState.<init>(ByteBufferDecodingState.java:44)
at org.apache.ozone.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:84)
at org.apache.hadoop.ozone.client.io.ECBlockReconstructedStripeInputStream.decodeStripe(ECBlockReconstructedStripeInputStream.java:649)
at org.apache.hadoop.ozone.client.io.ECBlockReconstructedStripeInputStream.read(ECBlockReconstructedStripeInputStream.java:388)
at org.apache.hadoop.ozone.client.io.TestECBlockReconstructedStripeInputStream.testErrorReadingBlockContinuesReading(TestECBlockReconstructedStripeInputStream.java:622)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
...
```
The solution is to set the limit on the buffer to the appropriate size before attempting to use it.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-7304
## How was this patch tested?
Modification to the unit test caused many test failures and then the code change fix them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel merged pull request #3817: HDDS-7304. EC: EC Decode can fail when byteBuffer from elastic pool is larger than chunksize
Posted by GitBox <gi...@apache.org>.
sodonnel merged PR #3817:
URL: https://github.com/apache/ozone/pull/3817
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org