You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sodonnel (via GitHub)" <gi...@apache.org> on 2024/03/15 11:20:18 UTC
[PR] HDDS-10527. KeyOverwrite with optimisitic locking [ozone]
sodonnel opened a new pull request, #6385:
URL: https://github.com/apache/ozone/pull/6385
## What changes were proposed in this pull request?
This change introduces the ability to re-create / overwrite a key in Ozone using an optimistic locking technique.
Say there is a desire to replace a key with some additional data added somewhere in the key, or perhaps change its replication type from Ratis to EC. To do this, you can read the current key data, write a new key with the same name, and then on commitKey, the new key version will be visible.
However, there is a possibility that some other client deletes the original key, or re-writes it at the same time, resulting in potential lost updates.
To replace a key in this way, the proposal is to use the existing objectID and updateID on the key to ensure the key has not changed since it was read. The flow would be:
1. Get the keyInfo for the current key.
2. Call the new bucket.overWriteKey() method, passing the details of the existing key
3. This call will adjust the keyArgs to pass two new fields - overwriteObjectID and updateObjectID which are taken from the objectID and updateID of the existing key.
4. When OM receives the open key request, it checks that an existing key is present having the passed keyname, objectID and updateID. If not, an error is returned. Otherwise the key is added to the openKeyTable, storing the overwrite IDs.
5. The data is written to the key as usual.
6. On key commit, the values stored in the openKey table for the overwrite IDs are checked against the current key. If the current key is absent, or its IDs have changed, the commit will fail and an error is returned. Otherwise the key is committed as usual.
This technique is similar to optimistic locking used in relational databases, to avoid holding a lock on an object for a long period of time.
Notably there are no additional locks needed on OM and no additional calls or rocksDB reads required to implement this - passing and storing the IDs in the openKey table is all that is required. The overwriteIDs don't need to be stored in the keyTable.
This change only added the feature for Object Store buckets for now.
Additionally, there is a question over what to do about meta-data and ACLs. Should they be copied from the existing key, or passed from the client.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10527
## How was this patch tested?
New integration and unit tests added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1538360815
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRequest.java:
##########
@@ -775,10 +775,19 @@ protected OmKeyInfo prepareFileInfo(
dbKeyInfo.setReplicationConfig(replicationConfig);
// Construct a new metadata map from KeyArgs.
- // Clear the old one when the key is overwritten.
- dbKeyInfo.getMetadata().clear();
- dbKeyInfo.getMetadata().putAll(KeyValueUtil.getFromProtobuf(
- keyArgs.getMetadataList()));
+ // Clear the old one when the key is overwritten unless it is a key overwrite
+ // using optimistic commit.
+ if (!keyArgs.hasOverwriteUpdateID()) {
+ dbKeyInfo.getMetadata().clear();
+ dbKeyInfo.getMetadata().putAll(KeyValueUtil.getFromProtobuf(
+ keyArgs.getMetadataList()));
+ }
+ if (keyArgs.hasOverwriteObjectID()) {
+ dbKeyInfo.setOverwriteObjectID(keyArgs.getOverwriteObjectID());
+ }
+ if (keyArgs.hasOverwriteUpdateID()) {
+ dbKeyInfo.setOverwriteUpdateID(keyArgs.getOverwriteUpdateID());
+ }
Review Comment:
The original code is confusing because it copies the existing key and then modifies into the new key that gets aded to the open key table, while leaving the original in the DB. I checked the fields being modified in the original and it looks correct, including preserving the ACLs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2032464337
> > @sodonnel If there is snapshot, then having overwrite will actually increase size, as there will be 2 copy - 1 - snapshot and 2 - active db with this solution.
>
> Correct. This feature, at least initially will only change the latest version of a key. If the key is under a snapshot it will cause the data to be duplicated.
@sodonnel
IMO, another impacts,
- When snapshot is deleted, that key blocks also needs to be deleted. Since no delete event, this will cause orphan blocks (where activeDB key is already overwritten). Need verify if implicitly handled or not (like another client does key overwrite with new set of blocks)
- quota size calculation -- Now key type gets changed from Ratis to EC or vice versa, so that impact for used size to be handled while calculating old and new to update difference.
- parallel operation -- before overwrite in commit, key gets deleted, so releasing blocks created for overwrite failure -- Do key is present in open key table for this scenario? so that on failure, and cleanup of open key, blocks are released?
Any reason of keeping same updateID ? as already we have key overwrite, same logic can follow from server side?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1595695455
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java:
##########
@@ -1431,6 +1412,59 @@ public OzoneOutputStream createKey(
return createOutputStream(openKey);
}
+ @Override
+ public OzoneOutputStream rewriteKey(String volumeName, String bucketName, String keyName,
+ long size, long existingKeyGeneration, ReplicationConfig replicationConfig,
+ Map<String, String> metadata) throws IOException {
+ createKeyPreChecks(volumeName, bucketName, keyName, replicationConfig);
+ String ownerName = getRealUserInfo().getShortUserName();
Review Comment:
This needs some more thought. I am going to defer this to another followup Jira so we can work out how it should behave.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1595932038
##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCreateRequest.java:
##########
@@ -903,6 +924,80 @@ public void testKeyCreateInheritParentDefaultAcls(
}
+ @ParameterizedTest
+ @MethodSource("data")
+ public void testAtomicRewrite(
+ boolean setKeyPathLock, boolean setFileSystemPaths) throws Exception {
+ when(ozoneManager.getOzoneLockProvider()).thenReturn(
+ new OzoneLockProvider(setKeyPathLock, setFileSystemPaths));
+
+ if (getBucketLayout() == BucketLayout.FILE_SYSTEM_OPTIMIZED) {
+ // TODO: Test is not applicable for FSO layout.
+ return;
+ }
Review Comment:
I am going to leave this as it is, as this is temporary until we implement the feature for FSO, which we will do on the branch before it is merged back to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2013951931
> As I first read your comment, I thought maybe I can cut this down to just checking the objectID
Actually I'm thinking we can just check the update ID to determine if the key changed. The case where object ID alone would be useful is if only a key's native ACLs were updated and we wanted to disregard this change. However, I think we want this API to preserve native ACLs in the original key. This makes it consistent with Ranger ACLs which will remain the same as long as the key is re-rewritten to the same location.
> but I think the append / hsync thing changes the picture slightly, needing the update ID to be checked too if we want to ensure there are no changes.
I don't think a key with an open lease being hsync'ed should be eligible for overwrite. I think this overwrite case is already blocked by existing hsync changes on the feature branch but we should double check to make sure we don't have issues later. I believe append will require an OM side update every time it is called, so yes that should increment the update ID.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2013968721
> I don't think a key with an open lease being hsync'ed should be eligible for overwrite
A key being hsync'ed is "open but visible" in Ozone and hence should have a lease which blocks other writers.
I think the "hsync / hbase" work also allows for a key to be appended - ie the key is closed and committed. Then a writer reopens it and appends some new data and commits / closes it again.
While the first scenario should be blocked, I need to ensure we do the right thing if an append happens to a closed key that is currently being overwrriten, and whether that is even possible! I will ask around about that.
I think you could be write that update_id is all we need as it will change on key append, hsync and object delete and recreate. It would be much nicer to only need to use 1 field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2103361069
@errose28 I believe I have addressed you further comments. Please have a look and let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1590731732
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java:
##########
@@ -74,11 +74,18 @@ public class OzoneKey {
* Constructs OzoneKey from OmKeyInfo.
*
*/
Review Comment:
This comment belongs to the constructor below the new member var.
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java:
##########
@@ -1671,7 +1708,8 @@ public List<OzoneKey> listKeys(String volumeName, String bucketName,
key.getModificationTime(),
key.getReplicationConfig(),
key.isFile(),
- key.getOwnerName()))
+ key.getOwnerName(),
+ key.getGeneration()))
Review Comment:
I guess `generation` will always be missing here. Any OM new enough to support atomic overwrite also has the `LIGHTWEIGHT_LIST_KEYS` feature, thus the first branch will be executed.
If we want to avoid adding generation to the lighter-weight key info used for listing, I think it's probably enough to return generation only for `getKeyDetails`, not for `listKeys`.
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java:
##########
@@ -1431,6 +1412,62 @@ public OzoneOutputStream createKey(
return createOutputStream(openKey);
}
+ @Override
+ public OzoneOutputStream rewriteKey(String volumeName, String bucketName, String keyName,
+ long size, long existingKeyGeneration, ReplicationConfig replicationConfig,
+ Map<String, String> metadata) throws IOException {
+ if (keyName == null) {
+ throw new IllegalArgumentException("Key cannot be null");
+ }
Review Comment:
This is also checked in `createKeyPreChecks`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1594935417
##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCommitRequest.java:
##########
@@ -218,6 +224,75 @@ public void testValidateAndUpdateCache() throws Exception {
omKeyInfo.getLatestVersionLocations().getLocationList());
}
+ @Test
+ public void testAtomicRewrite() throws Exception {
Review Comment:
Optional suggestion:
The large multi-scenario test methods used in the three request testing classes here would be clearer if each scenario was split into its own test case, like `testRewriteWhenKeyDeleted`, `testRewriteWhenKeyUpdated`, `testRewritePreservesMetadata` etc. These smaller units make it easier to see what is actually being tested, and what scenarios may have been missed.
That said there's technically nothing wrong with the way it is implemented here so you can disregard this if you think there would be code duplication in test setup or other factors that make this suboptimal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1534849499
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java:
##########
@@ -469,6 +469,25 @@ public OzoneOutputStream createKey(String key, long size,
.createKey(volumeName, name, key, size, replicationConfig, keyMetadata);
}
+ /**
+ * Overwrite an existing key using optimistic locking. The existingKey must exist in Ozone to allow
+ * the new key to be created with the same name. Additionally, the existingKey must not have been
+ * modified since the time its details were read. This is controlled by the objectID and updateID
+ * fields in the existingKey. If the key is replaced the objectID will change. If it has been updated
+ * (eg appended) the updateID will change. If either of these have changed since the existingKey
+ * was read, either the initial key create will fail, or the key will fail to commit after the data
+ * has been written.
Review Comment:
> This change only added the feature for Object Store buckets for now.
Ah looks like FSO is out of scope according to the PR description. We should call that out in the API documentation as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2013690253
> In Ozone, if you create a key, with objectID=5, and then create a new key of the same name, the objectID, I believe, remains the same, but the update_id is incremented.
Object ID is derived from the Ratis transaction index of the create request. See [the method](https://github.com/apache/ozone/blob/4da5a642243a42bba1aea6701d3d6b4b3a9462b4/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java#L2147) doing the calculation that is called from all create requests. Note the extra space created around each object ID to account for directories that might need to be created with a file create request. **Creating a new key with the same name will have a different object ID.**
Update ID is also set on key create but additionally changed when metadata like Native ACLs are changed. This is just the Ratis transaction index with no additional modifications. See [this method](https://github.com/apache/ozone/blob/b537a6a710e83a413cfba87c02e0357c0365d114/hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/WithObjectID.java#L69) and everywhere it is used for what operations modify the update ID. I'm not sure what you mean "update ID is incremented". The update ID of the new key will be arbitrarily larger than the old key since it is a Ratis transaction index, but its value is not related to the update ID of the previous version of the key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2014163844
Can this be done in a feature branch instead of the master? Right now there is no use of this change and it does not work for FSO and needs more testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2010323242
> Why not just introduce an object generation field and expose this API (in the future for applications itself). Reference: https://cloud.google.com/storage/docs/json_api/v1/objects/update
> Parameter Name Value Description
> **Optional query parameters**
> generation long If present, selects a specific revision of this object (as opposed to the latest version, the default).
> ifGenerationMatch long Makes the operation conditional on whether the object's current generation matches the given value. Setting to 0 makes the operation succeed only if there are no live versions of the object.
> ifGenerationNotMatch long Makes the operation conditional on whether the object's current generation does not match the given value. If no live object exists, the precondition fails. Setting to 0 makes the operation succeed only if there is a live version of the object.
> ifMetagenerationMatch long Makes the operation conditional on whether the object's current metageneration matches the given value.
> ifMetagenerationNotMatch long Makes the operation conditional on whether the object's current metageneration does not match the given value.
> overrideUnlockedRetention boolean Applicable for object's that have an unlocked retention configuration: Required to be set to true if the operation includes a retention property that changes the mode to Locked, reduces the retainUntilTime, or removes the retention configuration from the object.
I think it would be possible to extend to this in the future, but it may need more thought about what all operations it impacts. Implementing this sort of generic approach probably needs to cover key delete and create (overwrite) at least. Perhaps move / rename too. We would need to think through what it would mean for FSO buckets (perhaps nothing different, but I have not given it deep thought).
The workings of the proposed solution here is hidden behind the new overwriteKey API on the bucket object, and that is the only external API I have changed. If we were able to get the generation concept added, the overwrite key could be changed to use it and further enhance this sort of feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1549976002
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java:
##########
@@ -469,6 +469,30 @@ public OzoneOutputStream createKey(String key, long size,
.createKey(volumeName, name, key, size, replicationConfig, keyMetadata);
}
+ /**
+ * Overwrite an existing key using optimistic locking. The existingKey must exist in Ozone to allow
+ * the new key to be created with the same name. Additionally, the existing Key must not have been
+ * modified since the time it's details were read. This is controlled by the updateID
+ * field in the existing Key. If the key is replaced or updated the updateID will change. If the
+ * updateID has changed since the existing Key was read, either the initial key create will fail,
+ * or the key will fail to commit after the data has been written as the checks are carried out
+ * both a key open and commit time.
+ *
+ * For now this feature only works on Object Store Buckets. FSO support will be added a later.
+ *
+ * @param existingKey Name of the key to be created.
+ * @param replicationConfig Replication configuration.
+ * @return OzoneOutputStream to which the data has to be written.
+ * @throws IOException
+ */
+ public OzoneOutputStream overwriteKey(OzoneKeyDetails existingKey, ReplicationConfig replicationConfig)
Review Comment:
```suggestion
public OzoneOutputStream atomicUpdateObject(UpdateID oldUpdateID, OzoneKeyDetails Key, ReplicationConfig replicationConfig)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2031995880
> @sodonnel If there is snapshot, then having overwrite will actually increase size, as there will be 2 copy - 1 - snapshot and 2 - active db with this solution.
Correct. This feature, at least initially will only change the latest version of a key. If the key is under a snapshot it will cause the data to be duplicated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2108804850
@kerneltime @errose28 Following up on the naming of the "expectedGeneration" field in the protobuf. Can you review my thoughts above so we can agree on what this should be called? This is the only outstanding item in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2104226746
@kerneltime @errose28 Can you please clarify exactly what you want it to be called in this PR, keeping in mind we don't have metaGeneration at the moment? Is it to be "generation", "dataGeneration", or perhaps "expectedGeneration", which makes sense when making a call that should only exceed when the given expected generation is present.
It seems strange to have simply "generation" to me, as ideally that would be the current generation of a key, which is currently "updateID", as we are reusing it. If we ever added metaGeneration, I doubt we would call it metaUpdateID, but perhaps we would to keep consistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1595537290
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java:
##########
@@ -343,6 +343,7 @@ private OzoneConsts() {
public static final String BUCKET_LAYOUT = "bucketLayout";
public static final String TENANT = "tenant";
public static final String USER_PREFIX = "userPrefix";
+ public static final String OVERWRITE_GENERATION = "overwriteGeneration";
Review Comment:
+1 on this, similarly for `OmKeyArgs.overwriteGeneration`. Although I haven't taken a look deeply, I think the "generation" concept can be reused in the future. For example, we can ensure follower read adheres to "read-your-own-writes" consistency by passing the generation that was just written to the follower to ensure that the value has been updated in the follower (instead of linearizable read which has higher latency since it requires the follower to contact the leader as well).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1535302109
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java:
##########
@@ -240,6 +240,13 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, TermIn
throw new OMException("Failed to " + action + " key, as " + dbOpenKey +
"entry is not found in the OpenKey table", KEY_NOT_FOUND);
}
+
+ validateOptimisticLockingOverwrite(keyToDelete, omKeyInfo, auditMap);
+ // Optimistic locking validation has passed. Now set the overwrite fields to null so they are
+ // not persisted in the key table.
+ omKeyInfo.setOverwriteUpdateID(null);
+ omKeyInfo.setOverwriteObjectID(null);
Review Comment:
Adding the overwrite IDs to the commit phase is much more difficult as the commit is performed by various sub-classes on the client. Eg Ratis vs EC vs any new write class. I did look into added them only to the commit, but concluded it was much easier to persist them to the open key table.
Therefore I prefer to keep it the way it is now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1535297523
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRequest.java:
##########
@@ -775,10 +775,19 @@ protected OmKeyInfo prepareFileInfo(
dbKeyInfo.setReplicationConfig(replicationConfig);
// Construct a new metadata map from KeyArgs.
- // Clear the old one when the key is overwritten.
- dbKeyInfo.getMetadata().clear();
- dbKeyInfo.getMetadata().putAll(KeyValueUtil.getFromProtobuf(
- keyArgs.getMetadataList()));
+ // Clear the old one when the key is overwritten unless it is a key overwrite
+ // using optimistic commit.
+ if (!keyArgs.hasOverwriteUpdateID()) {
+ dbKeyInfo.getMetadata().clear();
+ dbKeyInfo.getMetadata().putAll(KeyValueUtil.getFromProtobuf(
+ keyArgs.getMetadataList()));
+ }
+ if (keyArgs.hasOverwriteObjectID()) {
+ dbKeyInfo.setOverwriteObjectID(keyArgs.getOverwriteObjectID());
+ }
+ if (keyArgs.hasOverwriteUpdateID()) {
+ dbKeyInfo.setOverwriteUpdateID(keyArgs.getOverwriteUpdateID());
+ }
Review Comment:
I will check about the ACLs. However I am pretty sure the ACLs and all other meta data is automatically copied from the original key, as the "openKey" entry it take from the original closedKey entry. I will double check on that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1585167879
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java:
##########
@@ -469,6 +469,30 @@ public OzoneOutputStream createKey(String key, long size,
.createKey(volumeName, name, key, size, replicationConfig, keyMetadata);
}
+ /**
+ * Overwrite an existing key using optimistic locking. The existingKey must exist in Ozone to allow
+ * the new key to be created with the same name. Additionally, the existing Key must not have been
+ * modified since the time it's details were read. This is controlled by the updateID
+ * field in the existing Key. If the key is replaced or updated the updateID will change. If the
+ * updateID has changed since the existing Key was read, either the initial key create will fail,
+ * or the key will fail to commit after the data has been written as the checks are carried out
+ * both a key open and commit time.
+ *
+ * For now this feature only works on Object Store Buckets. FSO support will be added a later.
+ *
+ * @param existingKey Name of the key to be created.
+ * @param replicationConfig Replication configuration.
+ * @return OzoneOutputStream to which the data has to be written.
+ * @throws IOException
+ */
+ public OzoneOutputStream overwriteKey(OzoneKeyDetails existingKey, ReplicationConfig replicationConfig)
+ throws IOException {
+ if (this.bucketLayout != BucketLayout.OBJECT_STORE) {
+ throw new IllegalArgumentException("Optimistic locking is only supported on Object Store Buckets");
Review Comment:
We are going to do this on a branch now and plan to immediately work on the FSO implementation so I will remove this check entirely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2010288136
Why not just introduce an object generation field and expose this API (in the future for applications itself).
Reference: https://cloud.google.com/storage/docs/json_api/v1/objects/update
| Parameter Name | Value | Description |
|---|---|---|
| **Optional query parameters** | | |
| generation | long | If present, selects a specific revision of this object (as opposed to the latest version, the default). |
| ifGenerationMatch | long | Makes the operation conditional on whether the object's current generation matches the given value. Setting to 0 makes the operation succeed only if there are no live versions of the object. |
| ifGenerationNotMatch | long | Makes the operation conditional on whether the object's current generation does not match the given value. If no live object exists, the precondition fails. Setting to 0 makes the operation succeed only if there is a live version of the object. |
| ifMetagenerationMatch | long | Makes the operation conditional on whether the object's current metageneration matches the given value. |
| ifMetagenerationNotMatch | long | Makes the operation conditional on whether the object's current metageneration does not match the given value. |
| overrideUnlockedRetention | boolean | Applicable for object's that have an unlocked retention configuration: Required to be set to true if the operation includes a retention property that changes the mode to Locked, reduces the retainUntilTime, or removes the retention configuration from the object. |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2010333682
Create https://issues.apache.org/jira/browse/HDDS-10558 for the idea suggested by @kerneltime as it is something that is worth exploring, but probably needs a bit more design work than just this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2012119946
@devmadhuu Thanks for the review. I added one more commit that adds details to the audits.
@kerneltime In google cloud, they use the generationID to access versions of an object like:
```
gs://bucket/object#generation_ID
```
So there is surely some overlap with the Ozone version and how AWS does things. Objects that are not versioned also have a generationID, so its not tied to versioning, but it is used in versioning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1592721391
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java:
##########
@@ -74,11 +74,18 @@ public class OzoneKey {
* Constructs OzoneKey from OmKeyInfo.
*
*/
Review Comment:
Sorry, I didn't notice it was misplaced even before the patch. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2100989431
@errose28 @kerneltime @adoroszlai have you got any further comments on this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1595698373
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java:
##########
@@ -1431,6 +1412,59 @@ public OzoneOutputStream createKey(
return createOutputStream(openKey);
}
+ @Override
+ public OzoneOutputStream rewriteKey(String volumeName, String bucketName, String keyName,
+ long size, long existingKeyGeneration, ReplicationConfig replicationConfig,
+ Map<String, String> metadata) throws IOException {
+ createKeyPreChecks(volumeName, bucketName, keyName, replicationConfig);
+ String ownerName = getRealUserInfo().getShortUserName();
Review Comment:
https://issues.apache.org/jira/browse/HDDS-10656 raised for this under the epic for the branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. Rewrite key atomically [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel merged PR #6385:
URL: https://github.com/apache/ozone/pull/6385
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1595700703
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java:
##########
@@ -343,6 +343,7 @@ private OzoneConsts() {
public static final String BUCKET_LAYOUT = "bucketLayout";
public static final String TENANT = "tenant";
public static final String USER_PREFIX = "userPrefix";
+ public static final String OVERWRITE_GENERATION = "overwriteGeneration";
Review Comment:
This constant is only used in the audit log. I think it makes sense to call it overwrite (or probably rewrite) generation to make it distinct from the current key generation. Calling the rewriteGeneration just generation could be confusing if we have a current generation called just "generation" and then this atomic rewrite parameter called "generation".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2036011365
@sodonnel I think it would help if you were to write up a markdown document summarizing the new API and contribute that as its own PR that people could read and comment as an easier medium for discussion. A lot of the discussion happening on this PR is not related to the code, but to higher level design decisions that were made like ACL handling, tracking ID throughout the process, and the API name and spec itself. There may be good reason for doing it this way, but for myself and others in the community both now and in the future, it would be best to write out the options that were considered and why a given approach was chosen, rather than submitting code that has already made a lot of these decisions without showing the process to get there.
Some things that might be helpful to write out:
- What does the new API look like from the client side
- How will clients use the new API
- How does the API compare to what other storage systems offer? Are there any ideas there we can re-use?
- General correctness guarantees in terms of metadata and data
- What key metadata is preserved or updated at the time of create and commit
More generally I hope this extra step can become a normal part of any change having moderate impact, like a new client API we must support indefinitely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1549976002
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java:
##########
@@ -469,6 +469,30 @@ public OzoneOutputStream createKey(String key, long size,
.createKey(volumeName, name, key, size, replicationConfig, keyMetadata);
}
+ /**
+ * Overwrite an existing key using optimistic locking. The existingKey must exist in Ozone to allow
+ * the new key to be created with the same name. Additionally, the existing Key must not have been
+ * modified since the time it's details were read. This is controlled by the updateID
+ * field in the existing Key. If the key is replaced or updated the updateID will change. If the
+ * updateID has changed since the existing Key was read, either the initial key create will fail,
+ * or the key will fail to commit after the data has been written as the checks are carried out
+ * both a key open and commit time.
+ *
+ * For now this feature only works on Object Store Buckets. FSO support will be added a later.
+ *
+ * @param existingKey Name of the key to be created.
+ * @param replicationConfig Replication configuration.
+ * @return OzoneOutputStream to which the data has to be written.
+ * @throws IOException
+ */
+ public OzoneOutputStream overwriteKey(OzoneKeyDetails existingKey, ReplicationConfig replicationConfig)
Review Comment:
```suggestion
public OzoneOutputStream atomicUpdateObject(UpdateID oldUpdateID, OzoneKeyDetails Key, ReplicationConfig replicationConfig)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1549976002
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java:
##########
@@ -469,6 +469,30 @@ public OzoneOutputStream createKey(String key, long size,
.createKey(volumeName, name, key, size, replicationConfig, keyMetadata);
}
+ /**
+ * Overwrite an existing key using optimistic locking. The existingKey must exist in Ozone to allow
+ * the new key to be created with the same name. Additionally, the existing Key must not have been
+ * modified since the time it's details were read. This is controlled by the updateID
+ * field in the existing Key. If the key is replaced or updated the updateID will change. If the
+ * updateID has changed since the existing Key was read, either the initial key create will fail,
+ * or the key will fail to commit after the data has been written as the checks are carried out
+ * both a key open and commit time.
+ *
+ * For now this feature only works on Object Store Buckets. FSO support will be added a later.
+ *
+ * @param existingKey Name of the key to be created.
+ * @param replicationConfig Replication configuration.
+ * @return OzoneOutputStream to which the data has to be written.
+ * @throws IOException
+ */
+ public OzoneOutputStream overwriteKey(OzoneKeyDetails existingKey, ReplicationConfig replicationConfig)
Review Comment:
```suggestion
public OzoneOutputStream atomicUpdateObject(UpdateID oldUpdateID, OzoneKeyDetails existingKey, ReplicationConfig replicationConfig)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2035638173
@kerneltime After a discussion we had, you suggested the OzoneBucket.overwriteKey(...) API does not seem correct. I agree that the Java doc and the method name are not good and should be changed. However the way the API works is in the same pattern as the existing similar methods.
For example createKey looks like:
```
public OzoneOutputStream createKey(String key, long size,
ReplicationConfig replicationConfig,
Map<String, String> keyMetadata)
throws IOException {
return proxy
.createKey(volumeName, name, key, size, replicationConfig, keyMetadata);
}
```
Note it takes the keyName and repConfig and returns an output stream.
The way the new API is intended to be used, is to replace an existing key with a new replication type. To do that, the user must first get the OzoneKeyDetails of the existing key, which will container the updateID along with the keyName etc. Therefore the new API allows those key details to be passed in directly.
```
OzoneKeyDetails exisitingKey = bucket.getKey(keyName);
try (OutputStream os = bucket.overwriteKey(existingKey, newRepConfig) {
os.write(bucket.readKey(keyName))
}
```
To me this makes perfect sense as a usable API without any extra manipulation of the key read. It is pretty much in the same pattern as the the other existing APIs, which was my intention.
It would be possible to change the parameter list to something like:
```
bucket.atomicallyOverwriteKey(String keyName, long existingKeyUpdateID, ReplicationConfig newRepConfig)
```
But I actually think that is worse to use, as you then have to extract pieces from the keyInfo you loaded. And it directly exposes the updateID, which at the moment is hidden. Perhaps we want to expose it, but perhaps not at this stage. I may also be possible to overload the API to have both options.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2040287373
Added a design doc via PR #6482.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1535349852
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRequest.java:
##########
@@ -775,10 +775,19 @@ protected OmKeyInfo prepareFileInfo(
dbKeyInfo.setReplicationConfig(replicationConfig);
// Construct a new metadata map from KeyArgs.
- // Clear the old one when the key is overwritten.
- dbKeyInfo.getMetadata().clear();
- dbKeyInfo.getMetadata().putAll(KeyValueUtil.getFromProtobuf(
- keyArgs.getMetadataList()));
+ // Clear the old one when the key is overwritten unless it is a key overwrite
+ // using optimistic commit.
+ if (!keyArgs.hasOverwriteUpdateID()) {
+ dbKeyInfo.getMetadata().clear();
+ dbKeyInfo.getMetadata().putAll(KeyValueUtil.getFromProtobuf(
+ keyArgs.getMetadataList()));
+ }
+ if (keyArgs.hasOverwriteObjectID()) {
+ dbKeyInfo.setOverwriteObjectID(keyArgs.getOverwriteObjectID());
+ }
+ if (keyArgs.hasOverwriteUpdateID()) {
+ dbKeyInfo.setOverwriteUpdateID(keyArgs.getOverwriteUpdateID());
+ }
Review Comment:
I was also not sure why the meta data is explicitly cleared here in the original code, while other key attributes are not. The metadata here, is a map of key value pairs that can be attached to a key. For the overwrite case, we just want to keep whatever is there hence the change here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2014694683
> Can this be done in a feature branch instead of the master? Right now there is no use of this change and it does not work for FSO and needs more testing.
I am pretty strongly against feature branches, because:
1. Nobody reviews changes landing in them except the few people working on them.
2. They are painful to merge to the main branch
3. They are painful to keep up to date with frequent merge conflicts.
4. Merging to the main branch causes a lot of change dumped in all at once.
5. Merging to main requires a vote for a changeset that frankly nobody ever reviews.
6. We have no release in sight for Ozone, so there is plenty of time to add any additional PRs and tests.
These changes will do no harm if nobody uses them. We also plan to add FSO support afterward, but this PR is large enough already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2086655224
@errose28 @kerneltime I have updated this PR to match what we discussed in the design doc PR.
The only things in the design doc that are not included here are:
1. Using the version framework in the client to detect an old server. As this is going to go on a branch, we will do that in a followup PR.
2. Have not considered failing an atomic write at a block boundary. That will be investigate later, and is somewhat nice to have at this stage.
I also plan to send this PR to a branch - it will not be committed to master. I need to create the branch and then see if we can re-point the PR at the branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1535303643
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java:
##########
@@ -179,6 +207,16 @@ public ReplicationConfig getReplicationConfig() {
return replicationConfig;
}
+ @JsonIgnore
+ public Long getObjectID() {
Review Comment:
I didn't want to change the output displayed by key info command. It can be argued either way whether they should be displayed or not I think, as they are kind of internal "not normal client facing things".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1538383070
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java:
##########
@@ -240,6 +240,13 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, TermIn
throw new OMException("Failed to " + action + " key, as " + dbOpenKey +
"entry is not found in the OpenKey table", KEY_NOT_FOUND);
}
+
+ validateOptimisticLockingOverwrite(keyToDelete, omKeyInfo, auditMap);
+ // Optimistic locking validation has passed. Now set the overwrite fields to null so they are
+ // not persisted in the key table.
+ omKeyInfo.setOverwriteUpdateID(null);
+ omKeyInfo.setOverwriteObjectID(null);
Review Comment:
> You could also argue, why persist any of the key meta data on key open (created date, ACLs, etc).
After looking this over a bit I think these are actually bugs that we need to fix:
- Create time:
This one is more of a debugging/auditing inconvenience. We set create time in the create phase before the object is visible, and then set modification time in the commit phase when the key is visible. Ideally for a key that has just been created, I would think ctime and mtime would be expected to be the same. However since both create and commit show up in the audit log I guess you could argue that the current implementation is correct as well.
- ACLs
This one looks more concerning. I haven't tested this yet, but it looks like the ACLs at the time of create are what are also committed to the final key, without checking if the key being replaced had ACL updates in the mean time. For example:
- key1 exists with acl1
- key1' is created at the same path as key1
- ACLs for key1 are updated to acl2 by another user/admin.
- key1' is committed with acl1 that was read at create time.
Now the ACLs have gone back in time without the admin or user intending to make this change.
Looking at it from this angle, the existing approach looks like it should be fixed to write metadata like ACLs and create time at the time of commit. Once these bugs are fixed, persisting the overwrite update ID at the time of create does not make much sense in context either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2013745364
@errose28 I think that for a HSync (HBase related changes), when a key is appended or synced, the update ID should be increased while the objectID stays the same. As I first read your comment, I thought maybe I can cut this down to just checking the objectID, but I think the append / hsync thing changes the picture slightly, needing the update ID to be checked too if we want to ensure there are no changes.
By "update ID is incremented" I just meant it was changed. Not that it is increased by 1. I was aware that both IDs are derived from the Ratis Transaction ID.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1592766399
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java:
##########
@@ -1671,7 +1708,8 @@ public List<OzoneKey> listKeys(String volumeName, String bucketName,
key.getModificationTime(),
key.getReplicationConfig(),
key.isFile(),
- key.getOwnerName()))
+ key.getOwnerName(),
+ key.getGeneration()))
Review Comment:
It would be good to include the generation in listKeys too, but perhaps it is not necessary right now. In the use cases we are thinking of, the user would need to call getKeyDetails to read the key data anyway.
I will remove this for now, and added another Jira (HDDS-10826) to the epic to consider adding this to ListKeys(and the Light version), as it might be a slightly large extra change in this PR right now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1592795031
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java:
##########
@@ -74,11 +74,18 @@ public class OzoneKey {
* Constructs OzoneKey from OmKeyInfo.
*
*/
Review Comment:
I didn't even check that - but I did wonder how I managed to move it from one end of the file to the other.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1595683152
##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCommitRequest.java:
##########
@@ -218,6 +224,75 @@ public void testValidateAndUpdateCache() throws Exception {
omKeyInfo.getLatestVersionLocations().getLocationList());
}
+ @Test
+ public void testAtomicRewrite() throws Exception {
+ if (getBucketLayout() == BucketLayout.FILE_SYSTEM_OPTIMIZED) {
+ // TODO - does not work with in FSO for now
+ return;
+ }
+
+ Table<String, OmKeyInfo> openKeyTable = omMetadataManager.getOpenKeyTable(BucketLayout.DEFAULT);
Review Comment:
This should actually use the method `getBucketLayout` as this test class is sub-classed for FSO and OBS buckets. As it was coded, it would have failed when we enabled FSO bucket testing but removing the temporary `if (getBucketLayout() == BucketLayout.FILE_SYSTEM_OPTIMIZED)` clause at line 229.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1595675742
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java:
##########
@@ -1097,6 +1097,56 @@ public void testPutKey() throws IOException {
}
}
+ @Test
+ public void testOverwriteKey() throws IOException {
Review Comment:
I have added https://issues.apache.org/jira/browse/HDDS-10839 to the epic to cover this. We can add them as part of the ongoing work on the branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1595950688
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java:
##########
@@ -343,6 +343,7 @@ private OzoneConsts() {
public static final String BUCKET_LAYOUT = "bucketLayout";
public static final String TENANT = "tenant";
public static final String USER_PREFIX = "userPrefix";
+ public static final String OVERWRITE_GENERATION = "overwriteGeneration";
Review Comment:
This was meant as a more general comment on the PR which probably wasn't clear. I think there are some places where `rewriteGeneration` makes sense, like the audit log here and the `KeyArgs` that are passed in by the client. Basically the places where the client is instructing the server to use the generation for rewrite purposes. However, when the value is stored or returned like in `OmKeyInfo` and the `KeyInfo` proto I think we should just call it `generation`. At that point the value could technically be used for anything, not just a rewrite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1547846561
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java:
##########
@@ -240,6 +240,13 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, TermIn
throw new OMException("Failed to " + action + " key, as " + dbOpenKey +
"entry is not found in the OpenKey table", KEY_NOT_FOUND);
}
+
+ validateOptimisticLockingOverwrite(keyToDelete, omKeyInfo, auditMap);
+ // Optimistic locking validation has passed. Now set the overwrite fields to null so they are
+ // not persisted in the key table.
+ omKeyInfo.setOverwriteUpdateID(null);
+ omKeyInfo.setOverwriteObjectID(null);
Review Comment:
Any change to the key, including ACLs should change the keys updateID, as that is what we are relying on to test if the key has changed. That would cause the optimistic commit to fail.
From the client code perspective, I think its easier to set the ACLs etc at the time of key open, as otherwise they need to be potentially passed down to various sub-classes. This also adds different testing paths for each type of key (Ratis, EC). That is why I am persisting the overwriteUpdateID in the openKey part. It should be passed as part of open key so we can check they key has no change since it was read and the key write starts. It costs basically nothing to persist it in the open key table and saves some complexity on commit.
If you think there are bugs around existing ACLs / Created Time / Modification time handling then please raise Jiras for them. We cannot change that as part of this PR, as it would not make sense in this context. A different PR should take care of that.
On the created time / modification time front - I think could be argued either way. What is the creation time of a file in Linux? You open the file, you write a series of bytes of several minutes and close the file. The ctime is probably the time the file was opened. The mtime is the time the file was last changed - they can easily be different. With Ozone its not as clear cut as the key traditionally has not been visible until it is committed. They the difference between ctime and mtime is really the time it took to write the bytes. After HBase allows uncommitted keys to be visible, the behavior is more like Linux and hence is quite possibly correct as it stands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1535347945
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java:
##########
@@ -240,6 +240,13 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, TermIn
throw new OMException("Failed to " + action + " key, as " + dbOpenKey +
"entry is not found in the OpenKey table", KEY_NOT_FOUND);
}
+
+ validateOptimisticLockingOverwrite(keyToDelete, omKeyInfo, auditMap);
+ // Optimistic locking validation has passed. Now set the overwrite fields to null so they are
+ // not persisted in the key table.
+ omKeyInfo.setOverwriteUpdateID(null);
+ omKeyInfo.setOverwriteObjectID(null);
Review Comment:
You could also argue, why persist any of the key meta data on key open (created date, ACLs, etc). I think persisting the overwriteID in the open key table keeps with the existing pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1535345790
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java:
##########
@@ -1643,7 +1685,9 @@ public List<OzoneKey> listKeys(String volumeName, String bucketName,
key.getCreationTime(),
key.getModificationTime(),
key.getReplicationConfig(),
- key.isFile()))
+ key.isFile(),
+ null,
+ null))
Review Comment:
Ah I missed that one when I was cleaning up the others. I have fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1535343737
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java:
##########
@@ -70,11 +70,19 @@ public class OzoneKey {
* Constructs OzoneKey from OmKeyInfo.
*
*/
+
+ /**
+ * The object and update ID of an existing key. These will be null
+ * if the key is not yet created on OM and read from OM.
Review Comment:
I reworded that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2014904327
@errose28 I have addressed all the comments except extending the tests to ensure the acls are copied over. I will try to do that later today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2032543220
@sumitagrawl This change is the same as any other key overwrite, only it will no go through (ie commit) unless the update IDs match - that is the only change. All the delete / quota handling is unchanged. The original key gets deleted if the bucket is not versioned in the exact same way as if someone writes the same key again.
In the case of snapshot, the original key is deleted, but the snapshot causes the blocks to be retained. When the snapshot is deleted, the original blocks get deleted too. Same behavior as now. This PR doesn't change that at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1594881613
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/WithObjectID.java:
##########
@@ -53,6 +53,13 @@ public final long getUpdateID() {
return updateID;
}
+ /**
+ * Returns the generation of the object. Note this is currently the same as updateID.
+ * @return long
+ */
+ public final long getGeneration() {
Review Comment:
Volume and bucket info also implement this interface. I guess it doesn't hurt, but is this really what we want? If we ever use a different source for generation it would not make sense as part of the ObjectID mixin.
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java:
##########
@@ -343,6 +343,7 @@ private OzoneConsts() {
public static final String BUCKET_LAYOUT = "bucketLayout";
public static final String TENANT = "tenant";
public static final String USER_PREFIX = "userPrefix";
+ public static final String OVERWRITE_GENERATION = "overwriteGeneration";
Review Comment:
Should we just call this `generation`? We may use it for something other than overwrites in the future.
##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCommitRequest.java:
##########
@@ -218,6 +224,75 @@ public void testValidateAndUpdateCache() throws Exception {
omKeyInfo.getLatestVersionLocations().getLocationList());
}
+ @Test
+ public void testAtomicRewrite() throws Exception {
Review Comment:
Optional suggestion:
The large multi-scenario test methods used in the three request testing classes here would be clearer if each scenario was split into its own test case, like `testRewriteWhenKeyDeleted`, `testRewriteWhenKeyUpdated`, `testRewritePreservesMetadata` etc. These smaller units make it easier to see what is actually being tested, and what scenarios may have been missed.
That said there's technically nothing wrong with the way it is implemented here so you can disregard this if you think there would be code duplication in test setup of other factors that make this suboptimal.
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java:
##########
@@ -1784,8 +1806,7 @@ private void addKeyPrefixInfoToResultList(String keyPrefix,
keyInfo.getDataSize(), keyInfo.getCreationTime(),
keyInfo.getModificationTime(),
keyInfo.getReplicationConfig(),
- keyInfo.isFile(),
- keyInfo.getOwnerName());
+ keyInfo.isFile(), keyInfo.getOwnerName(), keyInfo.getUpdateID());
Review Comment:
Per [this comment](https://github.com/apache/ozone/pull/6385#discussion_r1592766399) I thought we were not including generation in the list key response? This relates to changes to `OzoneKey` too which looks to only be used for listing output.
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java:
##########
@@ -1431,6 +1412,59 @@ public OzoneOutputStream createKey(
return createOutputStream(openKey);
}
+ @Override
+ public OzoneOutputStream rewriteKey(String volumeName, String bucketName, String keyName,
+ long size, long existingKeyGeneration, ReplicationConfig replicationConfig,
+ Map<String, String> metadata) throws IOException {
+ createKeyPreChecks(volumeName, bucketName, keyName, replicationConfig);
+ String ownerName = getRealUserInfo().getShortUserName();
Review Comment:
Is the key owner expected to change to the user who rewrote the key? I'm thinking of a case where rewrite is done by a background process changing the replication type. All keys would get changed to be owned by whatever user that process runs as.
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java:
##########
@@ -1097,6 +1097,56 @@ public void testPutKey() throws IOException {
}
}
+ @Test
+ public void testOverwriteKey() throws IOException {
Review Comment:
It would be good to test end-to-end scenarios of rewrite interleaving with other operations here. They can be split into separate test cases since this class uses the same mini cluster instance for all tests, or a new class can be created if too many new tests are required.
Some e2e scenarios I can think of are:
- Key in OM is updated in between the initial read of generation and the rewrite request.
- Key in OM is updated between the when the rewrite request returns and the stream is closed (between the create and commit)
- Key in OM that was previously written using rewrite can be overwritten again without rewrite.
- The delete test here, but extracted to its own test for clarity, leaving this test only for the positive case.
##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCommitRequest.java:
##########
@@ -218,6 +224,75 @@ public void testValidateAndUpdateCache() throws Exception {
omKeyInfo.getLatestVersionLocations().getLocationList());
}
+ @Test
+ public void testAtomicRewrite() throws Exception {
+ if (getBucketLayout() == BucketLayout.FILE_SYSTEM_OPTIMIZED) {
+ // TODO - does not work with in FSO for now
+ return;
+ }
+
+ Table<String, OmKeyInfo> openKeyTable = omMetadataManager.getOpenKeyTable(BucketLayout.DEFAULT);
Review Comment:
`BucketLayout.DEFAULT` is `LEGACY`. Technically the same table but using `OBJECT_STORE` here would be clearer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1595537290
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java:
##########
@@ -343,6 +343,7 @@ private OzoneConsts() {
public static final String BUCKET_LAYOUT = "bucketLayout";
public static final String TENANT = "tenant";
public static final String USER_PREFIX = "userPrefix";
+ public static final String OVERWRITE_GENERATION = "overwriteGeneration";
Review Comment:
Agreed, and similarly for `OmKeyArgs.overwriteGeneration`. Although I haven't taken a look deeply, I think the "generation" concept can be reused in the future. For example, we can ensure follower read adheres to "read-your-own-writes" consistency by passing the generation that was just written to the follower to ensure that the value has been updated in the follower (instead of linearizable read which has higher latency since it requires the follower to contact the leader as well).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2103398402
Looking good overall, I think we just need to determine when to call it `rewriteGeneration` and when to just use `generation` from [this thread](https://github.com/apache/ozone/pull/6385#discussion_r1595950688). I still think we need one method used by all tests that defines metadata expectations before and after an overwrite as described in [this comment](https://github.com/apache/ozone/pull/6385#pullrequestreview-2046995945) but if you want to do a follow up PR for that I'm ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1538360815
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRequest.java:
##########
@@ -775,10 +775,19 @@ protected OmKeyInfo prepareFileInfo(
dbKeyInfo.setReplicationConfig(replicationConfig);
// Construct a new metadata map from KeyArgs.
- // Clear the old one when the key is overwritten.
- dbKeyInfo.getMetadata().clear();
- dbKeyInfo.getMetadata().putAll(KeyValueUtil.getFromProtobuf(
- keyArgs.getMetadataList()));
+ // Clear the old one when the key is overwritten unless it is a key overwrite
+ // using optimistic commit.
+ if (!keyArgs.hasOverwriteUpdateID()) {
+ dbKeyInfo.getMetadata().clear();
+ dbKeyInfo.getMetadata().putAll(KeyValueUtil.getFromProtobuf(
+ keyArgs.getMetadataList()));
+ }
+ if (keyArgs.hasOverwriteObjectID()) {
+ dbKeyInfo.setOverwriteObjectID(keyArgs.getOverwriteObjectID());
+ }
+ if (keyArgs.hasOverwriteUpdateID()) {
+ dbKeyInfo.setOverwriteUpdateID(keyArgs.getOverwriteUpdateID());
+ }
Review Comment:
The original code is confusing because it copies the existing key and then modifies into the new key that gets aded to the open key table, while leaving the original in the DB. I checked the fields being modified in the original and it looks correct, although I am unsure about whether preserving ACLs from create is really what we want in general.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2037162472
> * it, key gets deleted, so releasing blocks created for overwrite failure -- Do key is present in open key table for this scenario? so
@sodonnel Another query,
If overwrite happens, and old blocks are removed as cleanup. But client already in progress of reading blocks from datanode for big file. In this case, Client may get No_block exception from DN. Are we planning to handle client for this change to refresh metadata on this failure?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2040289465
@sumitagrawl The problem you describe is already present for overwriting a key, or indeed if the balancer moves a container, or if the OM cache is holding stale container locations. There is already a retry facility in the client which attempts to obtain new block locations in the event that the locations is has obtained are invalid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2010582541
> Create https://issues.apache.org/jira/browse/HDDS-10558 for the idea suggested by @kerneltime as it is something that is worth exploring, but probably needs a bit more design work than just this PR.
It would not make sense to have generation and also use objectID. It is essentially the same capability, one using just the objectID and another having a more developer friendly generation id. I would recommend to introduce generation ID here instead of using objectID. It would not make sense to expose object ID to developers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimisitic locking [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1530868087
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java:
##########
@@ -70,11 +70,19 @@ public class OzoneKey {
* Constructs OzoneKey from OmKeyInfo.
*
*/
+
+ /**
+ * The object and update ID of an existing key. These will be null
+ * if the key is not yet created on OM and read from OM.
+ */
+ private final Long objectID;
+ private final Long updateID;
+
@SuppressWarnings("parameternumber")
public OzoneKey(String volumeName, String bucketName,
String keyName, long size, long creationTime,
long modificationTime, ReplicationConfig replicationConfig,
- boolean isFile) {
+ boolean isFile, Long objectID, Long updateID) {
Review Comment:
Should we add the 2 extra parameters in an overloaded constructor? (to avoid breaking existing code and having to add `null, null` in various places) We can switch to using a builder separately.
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/protocol/ClientProtocol.java:
##########
@@ -353,6 +353,19 @@ OzoneOutputStream createKey(String volumeName, String bucketName,
Map<String, String> metadata)
throws IOException;
+ /**
+ * Overwrite an existing key using optimistic locking. The OzoneKeyDetails passed must contain
+ * the objectID and updateID of the key to be overwritten. The existing key must also exist on
+ * OM and have the same ObjectID and UpdateID as the one passed in or the key create / commit
+ * will fail.
+ * @param keyToOverwrite Existing key to overwrite
+ * @param replicationConfig The replication configuration for the new key
+ * @return {@link OzoneOutputStream}
+ * @throws IOException
+ */
+ OzoneOutputStream overWriteKey(OzoneKeyDetails keyToOverwrite, ReplicationConfig replicationConfig)
Review Comment:
```suggestion
OzoneOutputStream overwriteKey(OzoneKeyDetails keyToOverwrite, ReplicationConfig replicationConfig)
```
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java:
##########
@@ -469,6 +469,25 @@ public OzoneOutputStream createKey(String key, long size,
.createKey(volumeName, name, key, size, replicationConfig, keyMetadata);
}
+ /**
+ * Overwrite an existing key using optimistic locking. The existingKey must exist in Ozone to allow
+ * the new key to be created with the same name. Additionally, the existingKey must not have been
+ * modified since the time its details were read. This is controlled by the objectID and updateID
+ * fields in the existingKey. If the key is replaced the objectID will change. If it has been updated
+ * (eg appended) the updateID will change. If either of these have changed since the existingKey
+ * was read, either the initial key create will fail, or the key will fail to commit after the data
+ * has been written.
+ *
+ * @param existingKey Name of the key to be created.
+ * @param replicationConfig Replication configuration.
+ * @return OzoneOutputStream to which the data has to be written.
+ * @throws IOException
+ */
+ public OzoneOutputStream overWriteKey(OzoneKeyDetails existingKey, ReplicationConfig replicationConfig)
Review Comment:
nit: "overwrite" is used as one word in comments and some variable names, so I think method name should match that.
```suggestion
public OzoneOutputStream overwriteKey(OzoneKeyDetails existingKey, ReplicationConfig replicationConfig)
```
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java:
##########
@@ -1420,6 +1401,70 @@ public OzoneOutputStream createKey(
return createOutputStream(openKey);
}
+ @Override
+ public OzoneOutputStream overWriteKey(OzoneKeyDetails keyToOverwrite, ReplicationConfig replicationConfig)
+ throws IOException {
+ if (keyToOverwrite == null) {
+ throw new IllegalArgumentException("KeyToOverwrite cannot be null");
+ }
+ if (keyToOverwrite.getObjectID() == null) {
+ throw new IllegalArgumentException("KeyToOverwrite ObjectID cannot be null");
+ }
+ if (keyToOverwrite.getUpdateID() == null) {
+ throw new IllegalArgumentException("KeyToOverwrite UpdateID cannot be null");
+ }
+ createKeyPreChecks(keyToOverwrite.getVolumeName(), keyToOverwrite.getBucketName(),
+ keyToOverwrite.getName(), replicationConfig);
+
+ OmKeyArgs.Builder builder = new OmKeyArgs.Builder()
+ .setVolumeName(keyToOverwrite.getVolumeName())
+ .setBucketName(keyToOverwrite.getBucketName())
+ .setKeyName(keyToOverwrite.getName())
+ .setDataSize(keyToOverwrite.getDataSize())
+ .setReplicationConfig(replicationConfig)
+ .addAllMetadataGdpr(keyToOverwrite.getMetadata())
+ // TODO - if we are effectively cloning a key, probably the ACLs should
+ // be copied over server side. I am not too sure how this works.
+ .setAcls(getAclList())
Review Comment:
Good point. I think we should omit metadata and ACLs, if possible, and let OM copy them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2010740213
The generationID in google cloud appears to be like the version in AWS and in Ozone to some extent (I believe versioning is not fully implemented in Ozone).
From the docs:
> There is no guarantee that generation numbers increase for successive versions, only that each new version has a unique generation number.
> There is no relationship between the generation numbers of unrelated objects, even if the objects are in the same bucket.
From that, it is not clear if each generationID is unique across the bucket, or if two different keys can have the the same generationID.
In Ozone, if you create a key, with objectID=5, and then create a new key of the same name, the objectID, I believe, remains the same, but the update_id is incremented.
If you deleted the key (without versioning enabled) and recreated it, the object_ID will change.
UpdateID comes from the Ratis Transaction ID (if Ratis is enabled), so it is probably unique on its own without the objectID, but I am not sure if that can be trusted without Ratis enabled. Also based on comments in WithObjectID.java.
Therefore I think that to guarantee an object has not changed, we need both object and update ID.
For now, I don't think we should expose any of this via the S3 or Hadoop compatible filesystems. We could also tag the new bucket.overwriteKey() as experimental for now, giving us scope to remove it or change it later if we decide it is not useful. However I still think it could be adapted to a new approach easily behind the public interface. Basically, I think the whole Ozone version story and version ID along with object and update ID needs to be fully worked out before we expose anything more widely that is done in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2092568324
@errose28 @kerneltime, @adoroszlai has created the branch [HDDS-10656-atomic-key-overwrite](https://github.com/apache/ozone/tree/HDDS-10656-atomic-key-overwrite) and pointed this PR at that branch.
Please let us know if you are happy with the changes here, or if you have further comments so we can progress toward committing this onto the branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1571124162
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java:
##########
@@ -469,6 +469,30 @@ public OzoneOutputStream createKey(String key, long size,
.createKey(volumeName, name, key, size, replicationConfig, keyMetadata);
}
+ /**
+ * Overwrite an existing key using optimistic locking. The existingKey must exist in Ozone to allow
+ * the new key to be created with the same name. Additionally, the existing Key must not have been
+ * modified since the time it's details were read. This is controlled by the updateID
+ * field in the existing Key. If the key is replaced or updated the updateID will change. If the
+ * updateID has changed since the existing Key was read, either the initial key create will fail,
+ * or the key will fail to commit after the data has been written as the checks are carried out
+ * both a key open and commit time.
+ *
+ * For now this feature only works on Object Store Buckets. FSO support will be added a later.
+ *
+ * @param existingKey Name of the key to be created.
+ * @param replicationConfig Replication configuration.
+ * @return OzoneOutputStream to which the data has to be written.
+ * @throws IOException
+ */
+ public OzoneOutputStream overwriteKey(OzoneKeyDetails existingKey, ReplicationConfig replicationConfig)
Review Comment:
```suggestion
public OzoneOutputStream atomicUpdateKey(OzoneKeyDetails existingKey, ReplicationConfig replicationConfig)
```
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java:
##########
@@ -469,6 +469,30 @@ public OzoneOutputStream createKey(String key, long size,
.createKey(volumeName, name, key, size, replicationConfig, keyMetadata);
}
+ /**
+ * Overwrite an existing key using optimistic locking. The existingKey must exist in Ozone to allow
+ * the new key to be created with the same name. Additionally, the existing Key must not have been
+ * modified since the time it's details were read. This is controlled by the updateID
+ * field in the existing Key. If the key is replaced or updated the updateID will change. If the
+ * updateID has changed since the existing Key was read, either the initial key create will fail,
+ * or the key will fail to commit after the data has been written as the checks are carried out
+ * both a key open and commit time.
+ *
+ * For now this feature only works on Object Store Buckets. FSO support will be added a later.
+ *
+ * @param existingKey Name of the key to be created.
+ * @param replicationConfig Replication configuration.
+ * @return OzoneOutputStream to which the data has to be written.
+ * @throws IOException
+ */
+ public OzoneOutputStream overwriteKey(OzoneKeyDetails existingKey, ReplicationConfig replicationConfig)
+ throws IOException {
+ if (this.bucketLayout != BucketLayout.OBJECT_STORE) {
+ throw new IllegalArgumentException("Optimistic locking is only supported on Object Store Buckets");
Review Comment:
This validation should move into OM to allow older clients to be able to update keys when OM supports it.
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java:
##########
@@ -469,6 +469,30 @@ public OzoneOutputStream createKey(String key, long size,
.createKey(volumeName, name, key, size, replicationConfig, keyMetadata);
}
+ /**
+ * Overwrite an existing key using optimistic locking. The existingKey must exist in Ozone to allow
+ * the new key to be created with the same name. Additionally, the existing Key must not have been
+ * modified since the time it's details were read. This is controlled by the updateID
+ * field in the existing Key. If the key is replaced or updated the updateID will change. If the
+ * updateID has changed since the existing Key was read, either the initial key create will fail,
+ * or the key will fail to commit after the data has been written as the checks are carried out
+ * both a key open and commit time.
+ *
+ * For now this feature only works on Object Store Buckets. FSO support will be added a later.
Review Comment:
```suggestion
* This API allows to atomically update an existing key. The key read before invoking this API
* should remain unchanged for this key to be written. This is controlled by the updateID
* field in the existing Key param. If the key is replaced or updated the updateID will change. If the
* updateID has changed since the existing Key was read, either the initial key create will fail,
* or the key will fail to commit after the data has been written as the checks are carried out
* both a key open and commit time.
*
* For now this feature only works on Object Store Buckets. FSO support will be added a later.
```
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java:
##########
@@ -70,11 +70,19 @@ public class OzoneKey {
* Constructs OzoneKey from OmKeyInfo.
*
*/
+
+ /**
+ * The update ID of an existing key. This will be null if this OzoneKey
+ * object has not been created from an existing key read from OM, as OM allocates
+ * the update ID on commit of the key.
+ */
+ private final Long updateID;
Review Comment:
We are leaking internal names into public APIs. For public facing API, I would model it how google APIs have documented theirs : https://cloud.google.com/storage/docs/metadata#generation-number
(We do not have the split between metadata only updates vs. whole object.)
For now, I would change this to `generation` and document the guarantees that an application has around the use of generation. This should apply to the entire PR, where we use `updateID` which is a more nebulous name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2110705368
I have raised https://issues.apache.org/jira/browse/HDDS-10857 to decide on the naming of the passed generation in the API. Please comment on the Jira so we can decide on the naming.
As @adoroszlai has given a +1 and @errose28 voiced approval pending this last decision I will commit this PR onto the branch tomorrow. All the followup items are under the epic and will be taken care of before merging the branch to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1595694751
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/WithObjectID.java:
##########
@@ -53,6 +53,13 @@ public final long getUpdateID() {
return updateID;
}
+ /**
+ * Returns the generation of the object. Note this is currently the same as updateID.
+ * @return long
+ */
+ public final long getGeneration() {
Review Comment:
I have moved this down to the OMKeyInfo object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2030125300
@sodonnel If there is snapshot, then having overwrite will actually increase size, as there will be 2 copy - 1 - snapshot and 2 - active db with this solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2069175362
Converted to draft until #6482 is outstanding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1592707182
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java:
##########
@@ -1431,6 +1412,62 @@ public OzoneOutputStream createKey(
return createOutputStream(openKey);
}
+ @Override
+ public OzoneOutputStream rewriteKey(String volumeName, String bucketName, String keyName,
+ long size, long existingKeyGeneration, ReplicationConfig replicationConfig,
+ Map<String, String> metadata) throws IOException {
+ if (keyName == null) {
+ throw new IllegalArgumentException("Key cannot be null");
+ }
Review Comment:
Ah, well spotted. I must have missed removing that when I created the precheck method!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1592710099
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java:
##########
@@ -74,11 +74,18 @@ public class OzoneKey {
* Constructs OzoneKey from OmKeyInfo.
*
*/
Review Comment:
I think this belongs to the static method at the bottom of the class definition:
```
public static OzoneKey fromKeyInfo(OmKeyInfo keyInfo)
```
I have moved it there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2010687599
ObjectID is already there, as is updateID. They have not been introduced here, and they are already persisted and managed in OM. This PR only publicly exposes the new method on the bucket, with no intention of exposing it further.
If we are going to expose GenerationID through the APIs in the way the google cloud docs indicate, then we need to decide if that is something we want to start with. Eg Google cloud supports it, but AWS does not. Its more test surface, and more code to write and expose via all the interfaces and most importantly to be supported going forward. We would also have to worry about forward / backward compatibility if we are adding another ID to be persisted in OM. Old keys will not have a generationID, but perhaps it can be derived from the object / update ID. How does it tie in with the object Version? If it is a new ID to be stored, it will add a small storage / memory overhead on OM. I don't believe we should just implement something like that without giving it due consideration.
What is there in this PR, could easily be changed to use a generationID if it was introduce later and this feature sees some adoption as the use of object / updateID is contained and hidden from users of the API, which is why I would suggest exploring GenerationID in the other Jira I raised. I am trying to make the smallest useful change possible to allow for atomic key overwrite, without closing any paths for future improvements. Therefore I would like to move any design around GenerationID into the other Jira and move ahead with this one in its current direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2010960015
> The generationID in google cloud appears to be like the version in AWS and in Ozone to some extent (I believe versioning is not fully implemented in Ozone).
Versioning != generation ID. Generation is just a monotonically increasing number without preserving the older generations.
>
> From the docs:
>
> > There is no guarantee that generation numbers increase for successive versions, only that each new version has a unique generation number.
> > There is no relationship between the generation numbers of unrelated objects, even if the objects are in the same bucket.
>
> From that, it is not clear if each generationID is unique across the bucket, or if two different keys can have the the same generationID.
Each object has its own generation field, it is not based on the bucket.
>
> In Ozone, if you create a key, with objectID=5, and then create a new key of the same name, the objectID, I believe, remains the same, but the update_id is incremented.
>
> If you deleted the key (without versioning enabled) and recreated it, the object_ID will change.
>
> UpdateID comes from the Ratis Transaction ID (if Ratis is enabled), so it is probably unique on its own without the objectID, but I am not sure if that can be trusted without Ratis enabled. Also based on comments in WithObjectID.java.
>
> Therefore I think that to guarantee an object has not changed, we need both object and update ID.
>
> For now, I don't think we should expose any of this via the S3 or Hadoop compatible filesystems. We could also tag the new bucket.overwriteKey() as experimental for now, giving us scope to remove it or change it later if we decide it is not useful. However I still think it could be adapted to a new approach easily behind the public interface. Basically, I think the whole Ozone version story and version ID along with object and update ID needs to be fully worked out before we expose anything more widely that is done in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1534659562
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java:
##########
@@ -70,11 +70,19 @@ public class OzoneKey {
* Constructs OzoneKey from OmKeyInfo.
*
*/
+
+ /**
+ * The object and update ID of an existing key. These will be null
+ * if the key is not yet created on OM and read from OM.
Review Comment:
I'm not sure what this means. They will be null if you try to read a key that is not yet created? Should it say "These will be null if the key is not yet created on OM OR read from OM", i.e. they are only set when using the overwrite API?
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java:
##########
@@ -469,6 +469,25 @@ public OzoneOutputStream createKey(String key, long size,
.createKey(volumeName, name, key, size, replicationConfig, keyMetadata);
}
+ /**
+ * Overwrite an existing key using optimistic locking. The existingKey must exist in Ozone to allow
+ * the new key to be created with the same name. Additionally, the existingKey must not have been
+ * modified since the time its details were read. This is controlled by the objectID and updateID
+ * fields in the existingKey. If the key is replaced the objectID will change. If it has been updated
+ * (eg appended) the updateID will change. If either of these have changed since the existingKey
+ * was read, either the initial key create will fail, or the key will fail to commit after the data
+ * has been written.
Review Comment:
This needs to document file and directory rename handling too.
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneKey.java:
##########
@@ -179,6 +207,16 @@ public ReplicationConfig getReplicationConfig() {
return replicationConfig;
}
+ @JsonIgnore
+ public Long getObjectID() {
Review Comment:
Why are these json ignored? I think they would be good to see from an `ozone sh key info` command.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRequest.java:
##########
@@ -775,10 +775,19 @@ protected OmKeyInfo prepareFileInfo(
dbKeyInfo.setReplicationConfig(replicationConfig);
// Construct a new metadata map from KeyArgs.
- // Clear the old one when the key is overwritten.
- dbKeyInfo.getMetadata().clear();
- dbKeyInfo.getMetadata().putAll(KeyValueUtil.getFromProtobuf(
- keyArgs.getMetadataList()));
+ // Clear the old one when the key is overwritten unless it is a key overwrite
+ // using optimistic commit.
+ if (!keyArgs.hasOverwriteUpdateID()) {
+ dbKeyInfo.getMetadata().clear();
+ dbKeyInfo.getMetadata().putAll(KeyValueUtil.getFromProtobuf(
+ keyArgs.getMetadataList()));
+ }
+ if (keyArgs.hasOverwriteObjectID()) {
+ dbKeyInfo.setOverwriteObjectID(keyArgs.getOverwriteObjectID());
+ }
+ if (keyArgs.hasOverwriteUpdateID()) {
+ dbKeyInfo.setOverwriteUpdateID(keyArgs.getOverwriteUpdateID());
+ }
Review Comment:
I'm not sure I understand this section in the original code either. It looks like we are copying the key metadata as it exists at the time of create, when it seems we should actually be doing this at the time of commit. Wherever this logic is happening, this is probably where we want to copy the native ACLS for the overwrite case as well.
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java:
##########
@@ -1643,7 +1685,9 @@ public List<OzoneKey> listKeys(String volumeName, String bucketName,
key.getCreationTime(),
key.getModificationTime(),
key.getReplicationConfig(),
- key.isFile()))
+ key.isFile(),
+ null,
+ null))
Review Comment:
nit. We can use the constructor without these two parameters instead. Same result but looks cleaner here.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java:
##########
@@ -240,6 +240,13 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager, TermIn
throw new OMException("Failed to " + action + " key, as " + dbOpenKey +
"entry is not found in the OpenKey table", KEY_NOT_FOUND);
}
+
+ validateOptimisticLockingOverwrite(keyToDelete, omKeyInfo, auditMap);
+ // Optimistic locking validation has passed. Now set the overwrite fields to null so they are
+ // not persisted in the key table.
+ omKeyInfo.setOverwriteUpdateID(null);
+ omKeyInfo.setOverwriteObjectID(null);
Review Comment:
Do we even need these persisted to the open key table? The client could just hold the update ID in memory from the original read, and pass it as the overwrite update ID on commit for the OM to use only during the in-memory portion of the commit. If the client dies before finishing the overwrite it will lose its client ID and not be able to access the open key to resume writing anyways.
It seems like we can move all new logic to the commit phase without modifying key create requests or the protos that are written to the DB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "errose28 (via GitHub)" <gi...@apache.org>.
errose28 commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1534849499
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java:
##########
@@ -469,6 +469,25 @@ public OzoneOutputStream createKey(String key, long size,
.createKey(volumeName, name, key, size, replicationConfig, keyMetadata);
}
+ /**
+ * Overwrite an existing key using optimistic locking. The existingKey must exist in Ozone to allow
+ * the new key to be created with the same name. Additionally, the existingKey must not have been
+ * modified since the time its details were read. This is controlled by the objectID and updateID
+ * fields in the existingKey. If the key is replaced the objectID will change. If it has been updated
+ * (eg appended) the updateID will change. If either of these have changed since the existingKey
+ * was read, either the initial key create will fail, or the key will fail to commit after the data
+ * has been written.
Review Comment:
> This change only added the feature for Object Store buckets for now.
Ah looks like FSO is out of scope according to the PR description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "sodonnel (via GitHub)" <gi...@apache.org>.
sodonnel commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2104243435
> Looking good overall, I think we just need to determine when to call it `rewriteGeneration` and when to just use `generation` from [this thread](https://github.com/apache/ozone/pull/6385#discussion_r1595950688). I still think we need one method used by all tests that defines metadata expectations before and after an overwrite as described in [this comment](https://github.com/apache/ozone/pull/6385#pullrequestreview-2046995945) but if you want to do a follow up PR for that I'm ok.
I have created https://issues.apache.org/jira/browse/HDDS-10843 for the test improvements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1595537290
##########
hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java:
##########
@@ -343,6 +343,7 @@ private OzoneConsts() {
public static final String BUCKET_LAYOUT = "bucketLayout";
public static final String TENANT = "tenant";
public static final String USER_PREFIX = "userPrefix";
+ public static final String OVERWRITE_GENERATION = "overwriteGeneration";
Review Comment:
+1 on this, similarly for `OmKeyArgs.overwriteGeneration` (edit: `OmkeyArgs.overwriteGeneration` change can be ignored). Although I haven't taken a look deeply, I think the "generation" concept can be reused in the future. For example, we can ensure follower read adheres to "read-your-own-writes" consistency by passing the generation that was just written to the follower to ensure that the value has been updated in the follower (instead of linearizable read which has higher latency since it requires the follower to contact the leader as well).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #6385:
URL: https://github.com/apache/ozone/pull/6385#discussion_r1595605095
##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/key/TestOMKeyCreateRequest.java:
##########
@@ -903,6 +924,80 @@ public void testKeyCreateInheritParentDefaultAcls(
}
+ @ParameterizedTest
+ @MethodSource("data")
+ public void testAtomicRewrite(
+ boolean setKeyPathLock, boolean setFileSystemPaths) throws Exception {
+ when(ozoneManager.getOzoneLockProvider()).thenReturn(
+ new OzoneLockProvider(setKeyPathLock, setFileSystemPaths));
+
+ if (getBucketLayout() == BucketLayout.FILE_SYSTEM_OPTIMIZED) {
+ // TODO: Test is not applicable for FSO layout.
+ return;
+ }
Review Comment:
Nit: Maybe using `assumeTrue` might be better?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10527. KeyOverwrite with optimistic locking [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #6385:
URL: https://github.com/apache/ozone/pull/6385#issuecomment-2103481276
> Looking good overall, I think we just need to determine when to call it `rewriteGeneration` and when to just use `generation` from [this thread](https://github.com/apache/ozone/pull/6385#discussion_r1595950688). I still think we need one method used by all tests that defines metadata expectations before and after an overwrite as described in [this comment](https://github.com/apache/ozone/pull/6385#pullrequestreview-2046995945) but if you want to do a follow up PR for that I'm ok.
An object should have a `generation` when either metadata or data mutates it should change (ideally increment). We can add sub cases `dataGeneration` and `metadataGeneration`. Qualifying it as `rewriteGeneration` seems superfluous.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org