You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/03/02 18:17:13 UTC
[GitHub] [hadoop-ozone] hanishakoneru opened a new pull request #625:
HDDS-2980. Delete replayed entry from OpenKeyTable during commit
hanishakoneru opened a new pull request #625: HDDS-2980. Delete replayed entry from OpenKeyTable during commit
URL: https://github.com/apache/hadoop-ozone/pull/625
## What changes were proposed in this pull request?
During KeyCreate (and S3InitiateMultipartUpload), we do not check the OpenKeyTable if the key already exists. If it does exist and the transaction is replayed, we just override the key in OpenKeyTable. This is done to avoid extra DB reads.
During KeyCommit (or S3MultipartUploadCommit), if the key was already committed, then we do not replay the transaction. This would result in the OpenKeyTable entry to remain in the DB till it is garbage collected.
To avoid storing stale entries in OpenKeyTable, during commit replays, we should check the openKeyTable and delete the entry if it exists.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-2980
## How was this patch tested?
Will add unit test in next commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] hanishakoneru commented on a change in pull request
#625: HDDS-2980. Delete replayed entry from OpenKeyTable during commit
Posted by GitBox <gi...@apache.org>.
hanishakoneru commented on a change in pull request #625: HDDS-2980. Delete replayed entry from OpenKeyTable during commit
URL: https://github.com/apache/hadoop-ozone/pull/625#discussion_r396566755
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java
##########
@@ -183,21 +196,26 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager,
new CacheKey<>(dbOzoneKey),
new CacheValue<>(Optional.of(omKeyInfo), trxnLogIndex));
- omResponse.setCommitKeyResponse(CommitKeyResponse.newBuilder().build());
omClientResponse = new OMKeyCommitResponse(omResponse.build(),
- omKeyInfo, commitKeyRequest.getClientID());
+ omKeyInfo, dbOpenKey);
Review comment:
We are anyway computing the openKey using the clientID here. So instead of passing the clientID and computing that again in OMKeyCommitResponse, we can directly pass the openKey.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request
#625: HDDS-2980. Delete replayed entry from OpenKeyTable during commit
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #625: HDDS-2980. Delete replayed entry from OpenKeyTable during commit
URL: https://github.com/apache/hadoop-ozone/pull/625#discussion_r396189604
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java
##########
@@ -183,21 +196,26 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager,
new CacheKey<>(dbOzoneKey),
new CacheValue<>(Optional.of(omKeyInfo), trxnLogIndex));
- omResponse.setCommitKeyResponse(CommitKeyResponse.newBuilder().build());
omClientResponse = new OMKeyCommitResponse(omResponse.build(),
- omKeyInfo, commitKeyRequest.getClientID());
+ omKeyInfo, dbOpenKey);
Review comment:
We can still pass here clientID and generate openKey. Like how we used to generate open key before.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request
#625: HDDS-2980. Delete replayed entry from OpenKeyTable during commit
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #625: HDDS-2980. Delete replayed entry from OpenKeyTable during commit
URL: https://github.com/apache/hadoop-ozone/pull/625#discussion_r396581977
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java
##########
@@ -183,21 +196,26 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager,
new CacheKey<>(dbOzoneKey),
new CacheValue<>(Optional.of(omKeyInfo), trxnLogIndex));
- omResponse.setCommitKeyResponse(CommitKeyResponse.newBuilder().build());
omClientResponse = new OMKeyCommitResponse(omResponse.build(),
- omKeyInfo, commitKeyRequest.getClientID());
+ omKeyInfo, dbOpenKey);
Review comment:
I see that we are doing similar thing for ozone key, so that is why my question. Then we can do similar thing for ozone key also then?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] hanishakoneru commented on issue #625: HDDS-2980.
Delete replayed entry from OpenKeyTable during commit
Posted by GitBox <gi...@apache.org>.
hanishakoneru commented on issue #625: HDDS-2980. Delete replayed entry from OpenKeyTable during commit
URL: https://github.com/apache/hadoop-ozone/pull/625#issuecomment-606136196
Thanks for the review @bharatviswa504 .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] hanishakoneru commented on a change in pull request
#625: HDDS-2980. Delete replayed entry from OpenKeyTable during commit
Posted by GitBox <gi...@apache.org>.
hanishakoneru commented on a change in pull request #625: HDDS-2980. Delete replayed entry from OpenKeyTable during commit
URL: https://github.com/apache/hadoop-ozone/pull/625#discussion_r396592736
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java
##########
@@ -183,21 +196,26 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager,
new CacheKey<>(dbOzoneKey),
new CacheValue<>(Optional.of(omKeyInfo), trxnLogIndex));
- omResponse.setCommitKeyResponse(CommitKeyResponse.newBuilder().build());
omClientResponse = new OMKeyCommitResponse(omResponse.build(),
- omKeyInfo, commitKeyRequest.getClientID());
+ omKeyInfo, dbOpenKey);
Review comment:
We could. But since we have to pass the omKeyInfo irrespective of whether we pass ozoneKey or not, I didnt change that.
I am ok to have it either way. Will post an update.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request
#625: HDDS-2980. Delete replayed entry from OpenKeyTable during commit
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #625: HDDS-2980. Delete replayed entry from OpenKeyTable during commit
URL: https://github.com/apache/hadoop-ozone/pull/625#discussion_r398972414
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCommitPartRequest.java
##########
@@ -147,11 +146,6 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager,
throw new OMException("Failed to commit Multipart Upload key, as " +
openKey + "entry is not found in the openKey table",
KEY_NOT_FOUND);
- } else {
- // Check the OpenKeyTable if this transaction is a replay of ratis logs.
Review comment:
Not understood why this check is removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] hanishakoneru merged pull request #625: HDDS-2980.
Delete replayed entry from OpenKeyTable during commit
Posted by GitBox <gi...@apache.org>.
hanishakoneru merged pull request #625: HDDS-2980. Delete replayed entry from OpenKeyTable during commit
URL: https://github.com/apache/hadoop-ozone/pull/625
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] hanishakoneru commented on issue #625: HDDS-2980.
Delete replayed entry from OpenKeyTable during commit
Posted by GitBox <gi...@apache.org>.
hanishakoneru commented on issue #625: HDDS-2980. Delete replayed entry from OpenKeyTable during commit
URL: https://github.com/apache/hadoop-ozone/pull/625#issuecomment-602891210
Integration test failure (timeout) is unrelated and passes locally. Re-triggering the checks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] hanishakoneru commented on a change in pull request
#625: HDDS-2980. Delete replayed entry from OpenKeyTable during commit
Posted by GitBox <gi...@apache.org>.
hanishakoneru commented on a change in pull request #625: HDDS-2980. Delete replayed entry from OpenKeyTable during commit
URL: https://github.com/apache/hadoop-ozone/pull/625#discussion_r399554295
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCommitPartRequest.java
##########
@@ -147,11 +146,6 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager,
throw new OMException("Failed to commit Multipart Upload key, as " +
openKey + "entry is not found in the openKey table",
KEY_NOT_FOUND);
- } else {
- // Check the OpenKeyTable if this transaction is a replay of ratis logs.
Review comment:
This check was redundant. Irrespective of whether KeyCreate Request was replayed or not, if key+clientID exits in the openKey table, then the CommitPart request should also be executed (same as we do for KeyCommit Request).
If the same Key part was created again, the clientID would be different. Hence the openKey would also be different.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request
#625: HDDS-2980. Delete replayed entry from OpenKeyTable during commit
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #625: HDDS-2980. Delete replayed entry from OpenKeyTable during commit
URL: https://github.com/apache/hadoop-ozone/pull/625#discussion_r399699101
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCommitPartRequest.java
##########
@@ -147,11 +146,6 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager,
throw new OMException("Failed to commit Multipart Upload key, as " +
openKey + "entry is not found in the openKey table",
KEY_NOT_FOUND);
- } else {
- // Check the OpenKeyTable if this transaction is a replay of ratis logs.
Review comment:
Got it. thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org