You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Mark Gui (Jira)" <ji...@apache.org> on 2022/02/21 09:30:00 UTC
[jira] [Commented] (HDDS-6348) EC: PartialStripe failure handling logic is writing padding bytes also to DNs
[ https://issues.apache.org/jira/browse/HDDS-6348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17495419#comment-17495419 ]
Mark Gui commented on HDDS-6348:
--------------------------------
Ah, yes, I happen to hit the same problem too, the stripe rewrite logic on failure should not try to write the padding 0s as the normal flow in close() don't.
And it happens to report a similar trace as HDDS-6295, but they with different root causes:
{code:java}
2022-02-21 08:26:34,902 [pool-2-thread-4] ERROR freon.BaseFreonGenerator: Error on executing task 30
java.lang.IllegalArgumentException
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:130)
at org.apache.hadoop.ozone.client.io.ECKeyOutputStream.close(ECKeyOutputStream.java:534) <-- writeOffset != offset at org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:61)
at org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.lambda$createKey$36(OzoneClientKeyGenerator.java:150)
at com.codahale.metrics.Timer.time(Timer.java:101)
at org.apache.hadoop.ozone.freon.OzoneClientKeyGenerator.createKey(OzoneClientKeyGenerator.java:142)
at org.apache.hadoop.ozone.freon.BaseFreonGenerator.tryNextTask(BaseFreonGenerator.java:183)
at org.apache.hadoop.ozone.freon.BaseFreonGenerator.taskLoop(BaseFreonGenerator.java:163)
at org.apache.hadoop.ozone.freon.BaseFreonGenerator.lambda$startTaskRunners$1(BaseFreonGenerator.java:146)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) {code}
Here offset counts the padding bytes as well.
But when we commit the key we should have got the exact length of the key without the padding bytes.
> EC: PartialStripe failure handling logic is writing padding bytes also to DNs
> -----------------------------------------------------------------------------
>
> Key: HDDS-6348
> URL: https://issues.apache.org/jira/browse/HDDS-6348
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Priority: Major
>
> [~cchenaxchen] was chatting with me for an issue regarding offset mismatch failure.
> While I am reviewing the code, I figured out that, in retry logic we are writing the padding bytes as well. The reason is, we used the cached bytes limit as the length. But by that time cached bytes would have been updated with padding data while writing previous parity bytes.
> I am not very sure if this is the same issue, [~cchenaxchen] is facing, but let me fix this and we will verify with this fix it it can fix. If issue is same, I will change the reporter to [~cchenaxchen]
>
> I have a test to reproduce the case and I will upload it shortly.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org