You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/06/24 05:49:34 UTC
[GitHub] [ozone] rakeshadr opened a new pull request, #3544: HDDS-6934. OM crashed with OzoneManagerDoubleBuffer
rakeshadr opened a new pull request, #3544:
URL: https://github.com/apache/ozone/pull/3544
## What changes were proposed in this pull request?
Race condition between async thread `OzoneManagerDoubleBuffer.flushTransactions()` and BucketDeleteRequest:
**Case:**
While processing `OMKeysDeleteResponseWithFSO` by the `OzoneManagerDoubleBuffer.flushTransactions()` thread, response logic constructs "buck_Id" by querying BucketTable. Assume during this time, `DELETE_BUCKET` request for this bucket "buck_Id" has been processed and updated the DB. Now, `OMKeysDeleteResponseWithFSO` will not see the bucket and throws NPE.
**More Details:**
OzoneClient: deleteBucket with recursive flag = true. Now, ozone client will splits this user call to multiple RPC calls to OM.
step-1)List paths and deleteObjects.
step-2)Finally deleteBucket
**Preventive Action:**
Action-1) Revisited all the Response classes. VolumeId and bucketId should be passed to all *Response* objects before sending response back to the client. This will avoid fetch bucketId from DB again and avoids NPE.
Action-2) Validated bucketInfo and volumeInfo for the non-existent case and throws OMException.
long getVolumeId(String volume) throws IOException;
long getBucketId(String volume, String bucket) throws IOException;
```
2022-06-22 11:53:30,901 Deleted all the keys and responded back to the ozone client after adding an OMKeysDeleteResponseWithFSO entry to the OzoneManagerDoubleBuffer
2022-06-22 11:53:30,905 DeleteBucket call came for the "vol_Id/buck_Id" and responded back to ozone client after adding an OMKeysDeleteResponseWithFSO entry to the OzoneManagerDoubleBuffer
2022-06-22 11:53:30,907 OzoneManagerDoubleBuffer#flushTransactions kicked in and picks up OMKeysDeleteResponseWithFSO entry. While processing it tries to construct "vol_Id/buck_Id/prefixId" inorder to delete the path from DirTable & FileTable.
```
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6934
## How was this patch tested?
Added test cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] JyotinderSingh commented on a diff in pull request #3544: HDDS-6934. OM crashed with OzoneManagerDoubleBuffer
Posted by GitBox <gi...@apache.org>.
JyotinderSingh commented on code in PR #3544:
URL: https://github.com/apache/ozone/pull/3544#discussion_r906766847
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeysDeleteResponseWithFSO.java:
##########
@@ -44,14 +44,16 @@
public class OMKeysDeleteResponseWithFSO extends OMKeysDeleteResponse {
private List<OmKeyInfo> dirsList;
+ private long volumeId;
public OMKeysDeleteResponseWithFSO(
@NotNull OzoneManagerProtocolProtos.OMResponse omResponse,
@NotNull List<OmKeyInfo> keyDeleteList,
@NotNull List<OmKeyInfo> dirDeleteList, boolean isRatisEnabled,
- @NotNull OmBucketInfo omBucketInfo) {
+ @NotNull OmBucketInfo omBucketInfo, long volId) {
Review Comment:
I don't think we will have cases of this class being instantiated with a `null` value assigned to `volId` parameter due to the changes in `getVolumeId` and `getBucketId` methods. Should we annotate it with `@NonNull` as well? (Similarly for the other request classes as well)
```suggestion
@NotNull OmBucketInfo omBucketInfo, @NotNull long volId) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] rakeshadr commented on pull request #3544: HDDS-6934. OM crashed with OzoneManagerDoubleBuffer
Posted by GitBox <gi...@apache.org>.
rakeshadr commented on PR #3544:
URL: https://github.com/apache/ozone/pull/3544#issuecomment-1168818541
Thanks @JyotinderSingh for the reviews. I will merge it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] rakeshadr commented on a diff in pull request #3544: HDDS-6934. OM crashed with OzoneManagerDoubleBuffer
Posted by GitBox <gi...@apache.org>.
rakeshadr commented on code in PR #3544:
URL: https://github.com/apache/ozone/pull/3544#discussion_r907422294
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeysDeleteResponseWithFSO.java:
##########
@@ -44,14 +44,16 @@
public class OMKeysDeleteResponseWithFSO extends OMKeysDeleteResponse {
private List<OmKeyInfo> dirsList;
+ private long volumeId;
public OMKeysDeleteResponseWithFSO(
@NotNull OzoneManagerProtocolProtos.OMResponse omResponse,
@NotNull List<OmKeyInfo> keyDeleteList,
@NotNull List<OmKeyInfo> dirDeleteList, boolean isRatisEnabled,
- @NotNull OmBucketInfo omBucketInfo) {
+ @NotNull OmBucketInfo omBucketInfo, long volId) {
Review Comment:
Updated the patch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] rakeshadr merged pull request #3544: HDDS-6934. OM crashed with OzoneManagerDoubleBuffer
Posted by GitBox <gi...@apache.org>.
rakeshadr merged PR #3544:
URL: https://github.com/apache/ozone/pull/3544
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] rakeshadr commented on a diff in pull request #3544: HDDS-6934. OM crashed with OzoneManagerDoubleBuffer
Posted by GitBox <gi...@apache.org>.
rakeshadr commented on code in PR #3544:
URL: https://github.com/apache/ozone/pull/3544#discussion_r907422716
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeysDeleteResponseWithFSO.java:
##########
@@ -44,14 +44,16 @@
public class OMKeysDeleteResponseWithFSO extends OMKeysDeleteResponse {
private List<OmKeyInfo> dirsList;
+ private long volumeId;
public OMKeysDeleteResponseWithFSO(
@NotNull OzoneManagerProtocolProtos.OMResponse omResponse,
@NotNull List<OmKeyInfo> keyDeleteList,
@NotNull List<OmKeyInfo> dirDeleteList, boolean isRatisEnabled,
- @NotNull OmBucketInfo omBucketInfo) {
+ @NotNull OmBucketInfo omBucketInfo, long volId) {
Review Comment:
Thanks @JyotinderSingh for the comment. Could you please review the updated patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org