You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/07/23 01:09:26 UTC
[GitHub] [ozone] symious opened a new pull request, #3617: HDDS-7041. Avoid ConcurrentModificationException in RepeatedOmKeyInfo
symious opened a new pull request, #3617:
URL: https://github.com/apache/ozone/pull/3617
## What changes were proposed in this pull request?
The error is causing OM's crash. Should be caused by concurrent change on RepeatedOmKeyInfo's omKeyInfo list.
```
2022-07-22 15:20:39,215 [OMDoubleBufferFlushThread] ERROR org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer: Terminating with exit status 2: OMDoubleBuffer flush threadOMDoubleBufferFlushThreadencountered Throwable error
java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)
at java.util.ArrayList$Itr.next(ArrayList.java:859)
at org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo.getProto(RepeatedOmKeyInfo.java:70)
at org.apache.hadoop.ozone.om.codec.RepeatedOmKeyInfoCodec.toPersistedFormat(RepeatedOmKeyInfoCodec.java:50)
at org.apache.hadoop.ozone.om.codec.RepeatedOmKeyInfoCodec.toPersistedFormat(RepeatedOmKeyInfoCodec.java:35)
at org.apache.hadoop.hdds.utils.db.CodecRegistry.asRawData(CodecRegistry.java:85)
at org.apache.hadoop.hdds.utils.db.TypedTable.putWithBatch(TypedTable.java:130)
at org.apache.hadoop.ozone.om.response.key.OMKeyCommitResponse.updateDeletedTable(OMKeyCommitResponse.java:107)
at org.apache.hadoop.ozone.om.response.key.OMKeyCommitResponse.addToDBBatch(OMKeyCommitResponse.java:80)
at org.apache.hadoop.ozone.om.response.OMClientResponse.checkAndUpdateDB(OMClientResponse.java:68)
at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$2(OzoneManagerDoubleBuffer.java:256)
at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatchWithTrace(OzoneManagerDoubleBuffer.java:201)
at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$1(OzoneManagerDoubleBuffer.java:254)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:250)
at java.lang.Thread.run(Thread.java:748)
```
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-7041
## How was this patch tested?
Fixed crash in prod cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #3617: HDDS-7041. Avoid ConcurrentModificationException in RepeatedOmKeyInfo
Posted by GitBox <gi...@apache.org>.
symious commented on PR #3617:
URL: https://github.com/apache/ozone/pull/3617#issuecomment-1193490479
@adoroszlai Thanks for the advice. Updated the patch, please have a check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] symious commented on pull request #3617: HDDS-7041. Avoid ConcurrentModificationException in RepeatedOmKeyInfo
Posted by GitBox <gi...@apache.org>.
symious commented on PR #3617:
URL: https://github.com/apache/ozone/pull/3617#issuecomment-1193647377
> Note: Although we used to recommend pushing a new commit to re-trigger the tests, Github now allows committers to re-run only the failed jobs. This also prevents the case where previously passed checks fail in a new run.
@adoroszlai Thanks for the information. Could you help to re-run the failed jobs, don't have the privilege to rerun yet.
Thanks a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #3617: HDDS-7041. Avoid ConcurrentModificationException in RepeatedOmKeyInfo
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on PR #3617:
URL: https://github.com/apache/ozone/pull/3617#issuecomment-1193661072
@symious Sure, I'll re-run failed jobs. One caveat is that first we need to wait for all jobs to finish. (The other is that some checks depend on artifacts from the _build_ check, and these artifacts expire in 1 day.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai commented on pull request #3617: HDDS-7041. Avoid ConcurrentModificationException in RepeatedOmKeyInfo
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on PR #3617:
URL: https://github.com/apache/ozone/pull/3617#issuecomment-1193644984
Note: Although we used to recommend pushing a new commit to re-trigger the tests, Github now allows committers to re-run only the failed jobs. This also prevents the case where previously passed checks fail in a new run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] adoroszlai merged pull request #3617: HDDS-7041. Avoid ConcurrentModificationException in RepeatedOmKeyInfo
Posted by GitBox <gi...@apache.org>.
adoroszlai merged PR #3617:
URL: https://github.com/apache/ozone/pull/3617
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org