You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/08/30 01:57:05 UTC

[GitHub] [ozone] GeorgeJahad opened a new pull request, #3729: HDDS-6855. Path Based Access for Ozone Snapshots.

GeorgeJahad opened a new pull request, #3729:
URL: https://github.com/apache/ozone/pull/3729

## What changes were proposed in this pull request?
### OmSnapshot
This PR builds on the OmMReader PR https://github.com/apache/ozone/pull/3653 to create the new OmSnapshot abstraction which is an OmMReader that reads a snapshots metadata.

It's basically identical to the ozoneManager OmMReader with two exceptions:
1. Its keymanager and prefix manager contain an OmMetadataManager that reads from a snapshot instead of the active filesystem.
2. It normalizes/denormalizes each request as it comes in to remove/replace the ".snapshot/snapshotName" prefix.

### OmLock
The OmMetadataManager use locks to prevent the active filesystem from being corrupted. These locks are unnecessary for snapshotting, but I didn't want to modify all the metadatamanager code to check whether it is looking at a snapshot or not. Instead I created the OmLock interface with two implementations, the current one, and a new "read-only" lock for snapshots.

The new read only lock isn't a lock at all. It just returns true for aquiring read locks and false for write locks.

### OmSnapshotManager, (OSM)
The OmSnapshotManager contains the LRU cache of OmSnapshots.

When the OM gets a read request, it passes it to the OSM to see if it is a snapshot request. If so, the OSM retrieves the snapshot from the cache, (reading it in first if necessary,) and returns that snapshots OmMReader to the OM to complete the request.

If itt is not a snapshot, the OM just uses the standard OmMReader for the active filesystem.

### IOmMetadataReader interface
I also added this interface, (which is implemented by the existing OmMetadataReader. The OzoneManager and OmSnapshot also implement it and delegate the requests to their respective OmMetadataReader instances.) It wasn't strictly needed but seems cleaner to do it this way.

Let me know if you think it is overkill.

## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6855

## How was this patch tested?

There are many existing tests that exercise reading of the metadata. I wanted to leverage those as much as possible. So my basic approach has been to copy them into this PR and modify them just enough to support snapshots. The tests were originally structured something like this:

```
1. create some data
2. read back the data to confirm it was correctly created.
3. (possibly repeat the above steps to create more data.)
```

Between steps 1 and 2, I've added two more:
```
1a. create a snapshot
1b. delete the data from the active filesystem, (to be sure the snapshot code isn't inadvertently reading from there.)
```

In addition I've had to modify the read requests in the tests to use the snapshot prefix and read from the snapshot.

The original tests were taken from the TestOzoneFileSystem, TestOzoneRpcClientAbstract, and TestObjectStoreWithFSO test classes.

To be sure I've only changed the minimum required to support snapshotting, I've created a python script to generate the diffs between those original tests and my changes. Those diffs are [here](https://github.com/GeorgeJahad/ozone/compare/a27cc3729..977923847?w=1).

Where possible I've parameterized the tests so that they excercise all 3 bucket types.

## TODO
1. This PR handles most snapshot read operations with the exception of ACL's. That will be handled in the next PR.
2. We need to prevent users from writing to keys that are prefixed with ".snapshot", (because they won't be able to read them back out again). I added a [ticket](https://issues.apache.org/jira/browse/HDDS-7168) for it.

## UI
With this PR, the following types of commands now work:
```
ozone sh key get testgbj2/bucket1/.snapshot/snapshot1/k1 k2
ozone fs -ls /testgbj2/bucket1/.snapshot/snapshot1
ozone fs -cp /testgbj2/bucket1/.snapshot/snapshot1/k1 /testgbj2/bucket1/k3
```

These commands will init the snapshot so that the commands above can operate on it:
```
ozone sh volume create testgbj2
ozone sh bucket create testgbj2/bucket1
echo k1 > k1.orig
ozone sh key put testgbj2/bucket1/k1 k1.orig
ozone sh snapshot create testgbj2/bucket1 snapshot1
```

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org