You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/10/27 07:53:23 UTC
[GitHub] [ozone] jyotirmoy-gh opened a new pull request, #3896: Hdds 7427
jyotirmoy-gh opened a new pull request, #3896:
URL: https://github.com/apache/ozone/pull/3896
## What changes were proposed in this pull request?
Added unit-testcases for Ozone snapshot feature HDDS-6517
## What is the link to the Apache JIRA
[HDDS-7427](https://issues.apache.org/jira/browse/HDDS-7427)
## How was this patch tested?
Added testcases in file - hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshot.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sadanand48 commented on a diff in pull request #3896: HDDS-7427 - [snapshot] Add unit-testcases for Ozone snapshot feature.
Posted by GitBox <gi...@apache.org>.
sadanand48 commented on code in PR #3896:
URL: https://github.com/apache/ozone/pull/3896#discussion_r1006563012
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshot.java:
##########
@@ -383,6 +383,98 @@ public void checkKey() throws Exception {
snapshotKeyPrefix + key1);
}
+ @Test
+ public void testListDeleteKey()
+ throws IOException, InterruptedException, TimeoutException {
+ String volume = "vol-" + RandomStringUtils.randomNumeric(5);
+ String bucket = "buc-" + RandomStringUtils.randomNumeric(5);
+ store.createVolume(volume);
+ OzoneVolume vol = store.getVolume(volume);
+ vol.createBucket(bucket);
+ OzoneBucket volbucket = vol.getBucket(bucket);
+
+ String key = "key-";
+ byte[] value = RandomStringUtils.randomAscii(10240).getBytes(UTF_8);
+ OzoneOutputStream oneKey = volbucket.createKey(
+ key + RandomStringUtils.randomNumeric(5),
+ value.length, RATIS, ONE,
+ new HashMap<>());
+ oneKey.write(value);
+ oneKey.close();
+
+ String snapshotKeyPrefix = createSnapshot(volume, bucket);
+ Iterator<? extends OzoneKey> volBucketIter =
+ volbucket.listKeys(snapshotKeyPrefix + "key-");
+ int volBucketKeyCount = 0;
+ while (volBucketIter.hasNext()) {
+ volBucketIter.next();
+ volBucketKeyCount++;
+ }
+ Assert.assertEquals(1, volBucketKeyCount);
+
+ deleteKeys(volbucket);
+
+ snapshotKeyPrefix = createSnapshot(volume, bucket);
+ Iterator<? extends OzoneKey> volBucketIter2 =
+ volbucket.listKeys(snapshotKeyPrefix);
+ while (volBucketIter2.hasNext()) {
+ fail();
+ }
+ }
+
+ @Test
+ public void testListAddNewKey()
+ throws IOException, InterruptedException, TimeoutException {
+ String volume = "vol-" + RandomStringUtils.randomNumeric(5);
+ String bucket = "buc-" + RandomStringUtils.randomNumeric(5);
+ store.createVolume(volume);
+ OzoneVolume vol = store.getVolume(volume);
+ vol.createBucket(bucket);
+ OzoneBucket volbucket = vol.getBucket(bucket);
+
+ String key = "key-";
+ byte[] value = RandomStringUtils.randomAscii(10240).getBytes(UTF_8);
+ OzoneOutputStream oneKey = volbucket.createKey(
Review Comment:
There are many occurrences of this code to create and write to a key, we can maybe create a method for this to reduce duplicates.
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshot.java:
##########
@@ -383,6 +383,98 @@ public void checkKey() throws Exception {
snapshotKeyPrefix + key1);
}
+ @Test
+ public void testListDeleteKey()
+ throws IOException, InterruptedException, TimeoutException {
+ String volume = "vol-" + RandomStringUtils.randomNumeric(5);
+ String bucket = "buc-" + RandomStringUtils.randomNumeric(5);
+ store.createVolume(volume);
+ OzoneVolume vol = store.getVolume(volume);
+ vol.createBucket(bucket);
+ OzoneBucket volbucket = vol.getBucket(bucket);
+
+ String key = "key-";
+ byte[] value = RandomStringUtils.randomAscii(10240).getBytes(UTF_8);
+ OzoneOutputStream oneKey = volbucket.createKey(
+ key + RandomStringUtils.randomNumeric(5),
+ value.length, RATIS, ONE,
+ new HashMap<>());
+ oneKey.write(value);
+ oneKey.close();
+
+ String snapshotKeyPrefix = createSnapshot(volume, bucket);
+ Iterator<? extends OzoneKey> volBucketIter =
+ volbucket.listKeys(snapshotKeyPrefix + "key-");
+ int volBucketKeyCount = 0;
+ while (volBucketIter.hasNext()) {
+ volBucketIter.next();
+ volBucketKeyCount++;
+ }
+ Assert.assertEquals(1, volBucketKeyCount);
+
+ deleteKeys(volbucket);
Review Comment:
nit : It may be better to delete right after createSnapshot and then verify whether the snapshot has count of 1 key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on a diff in pull request #3896: HDDS-7427. [Snapshot] Add unit-testcases for Ozone snapshot feature.
Posted by GitBox <gi...@apache.org>.
smengcl commented on code in PR #3896:
URL: https://github.com/apache/ozone/pull/3896#discussion_r1007616957
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshot.java:
##########
@@ -383,6 +340,78 @@ public void checkKey() throws Exception {
snapshotKeyPrefix + key1);
}
+ @Test
+ public void testListDeleteKey()
+ throws IOException, InterruptedException, TimeoutException {
+ String volume = "vol-" + RandomStringUtils.randomNumeric(5);
+ String bucket = "buc-" + RandomStringUtils.randomNumeric(5);
+ store.createVolume(volume);
+ OzoneVolume vol = store.getVolume(volume);
+ vol.createBucket(bucket);
+ OzoneBucket volbucket = vol.getBucket(bucket);
+
+ String key = "key-";
+ createFileKey(volbucket, key);
+ String snapshotKeyPrefix = createSnapshot(volume, bucket);
+ deleteKeys(volbucket);
+
+ Iterator<? extends OzoneKey> volBucketIter =
+ volbucket.listKeys(snapshotKeyPrefix + "key-");
+ int volBucketKeyCount = 0;
+ while (volBucketIter.hasNext()) {
+ volBucketIter.next();
+ volBucketKeyCount++;
+ }
+ Assert.assertEquals(1, volBucketKeyCount);
+
+ snapshotKeyPrefix = createSnapshot(volume, bucket);
+ Iterator<? extends OzoneKey> volBucketIter2 =
+ volbucket.listKeys(snapshotKeyPrefix);
+ while (volBucketIter2.hasNext()) {
+ fail();
Review Comment:
nit: add a sensible message explaining what is the expected outcome to assist other dev when debugging this UT just in case it fails. e.g.
```suggestion
fail("The last snapshot should not have any keys in it!");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sadanand48 commented on a diff in pull request #3896: HDDS-7427 - [snapshot] Add unit-testcases for Ozone snapshot feature.
Posted by GitBox <gi...@apache.org>.
sadanand48 commented on code in PR #3896:
URL: https://github.com/apache/ozone/pull/3896#discussion_r1006915116
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshot.java:
##########
@@ -416,5 +488,16 @@ private void deleteKeys(OzoneBucket bucket) throws IOException {
bucket.deleteKey(key.getName());
}
}
+
+ private void createKeys(OzoneBucket bucket, String keyprefix) throws IOException {
Review Comment:
We can also use this in ` testListKey ` if possible. Also the method name can be `createKey` instead of `createKeys `since its writing a single key
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on pull request #3896: HDDS-7427. [Snapshot] Add unit-testcases for Ozone snapshot feature.
Posted by GitBox <gi...@apache.org>.
smengcl commented on PR #3896:
URL: https://github.com/apache/ozone/pull/3896#issuecomment-1295671799
Thanks @jyotirmoy-gh for the patch. Thanks @sadanand48 for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl merged pull request #3896: HDDS-7427. [Snapshot] Add unit-testcases for Ozone snapshot feature.
Posted by GitBox <gi...@apache.org>.
smengcl merged PR #3896:
URL: https://github.com/apache/ozone/pull/3896
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] jyotirmoy-gh commented on a diff in pull request #3896: HDDS-7427. [Snapshot] Add unit-testcases for Ozone snapshot feature.
Posted by GitBox <gi...@apache.org>.
jyotirmoy-gh commented on code in PR #3896:
URL: https://github.com/apache/ozone/pull/3896#discussion_r1006994682
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshot.java:
##########
@@ -416,5 +488,16 @@ private void deleteKeys(OzoneBucket bucket) throws IOException {
bucket.deleteKey(key.getName());
}
}
+
+ private void createKeys(OzoneBucket bucket, String keyprefix) throws IOException {
Review Comment:
I have renamed the method to createFileKey to avoid conflict with another method. Implemented the method in testListKey
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] jyotirmoy-gh commented on a diff in pull request #3896: HDDS-7427. [Snapshot] Add unit-testcases for Ozone snapshot feature.
Posted by GitBox <gi...@apache.org>.
jyotirmoy-gh commented on code in PR #3896:
URL: https://github.com/apache/ozone/pull/3896#discussion_r1007754619
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshot.java:
##########
@@ -383,6 +340,78 @@ public void checkKey() throws Exception {
snapshotKeyPrefix + key1);
}
+ @Test
+ public void testListDeleteKey()
+ throws IOException, InterruptedException, TimeoutException {
+ String volume = "vol-" + RandomStringUtils.randomNumeric(5);
+ String bucket = "buc-" + RandomStringUtils.randomNumeric(5);
+ store.createVolume(volume);
+ OzoneVolume vol = store.getVolume(volume);
+ vol.createBucket(bucket);
+ OzoneBucket volbucket = vol.getBucket(bucket);
+
+ String key = "key-";
+ createFileKey(volbucket, key);
+ String snapshotKeyPrefix = createSnapshot(volume, bucket);
+ deleteKeys(volbucket);
+
+ Iterator<? extends OzoneKey> volBucketIter =
+ volbucket.listKeys(snapshotKeyPrefix + "key-");
+ int volBucketKeyCount = 0;
+ while (volBucketIter.hasNext()) {
+ volBucketIter.next();
+ volBucketKeyCount++;
+ }
+ Assert.assertEquals(1, volBucketKeyCount);
+
+ snapshotKeyPrefix = createSnapshot(volume, bucket);
+ Iterator<? extends OzoneKey> volBucketIter2 =
+ volbucket.listKeys(snapshotKeyPrefix);
+ while (volBucketIter2.hasNext()) {
+ fail();
Review Comment:
Added failure debug message
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] jyotirmoy-gh commented on a diff in pull request #3896: HDDS-7427 - [snapshot] Add unit-testcases for Ozone snapshot feature.
Posted by GitBox <gi...@apache.org>.
jyotirmoy-gh commented on code in PR #3896:
URL: https://github.com/apache/ozone/pull/3896#discussion_r1006630151
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshot.java:
##########
@@ -383,6 +383,98 @@ public void checkKey() throws Exception {
snapshotKeyPrefix + key1);
}
+ @Test
+ public void testListDeleteKey()
+ throws IOException, InterruptedException, TimeoutException {
+ String volume = "vol-" + RandomStringUtils.randomNumeric(5);
+ String bucket = "buc-" + RandomStringUtils.randomNumeric(5);
+ store.createVolume(volume);
+ OzoneVolume vol = store.getVolume(volume);
+ vol.createBucket(bucket);
+ OzoneBucket volbucket = vol.getBucket(bucket);
+
+ String key = "key-";
+ byte[] value = RandomStringUtils.randomAscii(10240).getBytes(UTF_8);
+ OzoneOutputStream oneKey = volbucket.createKey(
+ key + RandomStringUtils.randomNumeric(5),
+ value.length, RATIS, ONE,
+ new HashMap<>());
+ oneKey.write(value);
+ oneKey.close();
+
+ String snapshotKeyPrefix = createSnapshot(volume, bucket);
+ Iterator<? extends OzoneKey> volBucketIter =
+ volbucket.listKeys(snapshotKeyPrefix + "key-");
+ int volBucketKeyCount = 0;
+ while (volBucketIter.hasNext()) {
+ volBucketIter.next();
+ volBucketKeyCount++;
+ }
+ Assert.assertEquals(1, volBucketKeyCount);
+
+ deleteKeys(volbucket);
+
+ snapshotKeyPrefix = createSnapshot(volume, bucket);
+ Iterator<? extends OzoneKey> volBucketIter2 =
+ volbucket.listKeys(snapshotKeyPrefix);
+ while (volBucketIter2.hasNext()) {
+ fail();
+ }
+ }
+
+ @Test
+ public void testListAddNewKey()
+ throws IOException, InterruptedException, TimeoutException {
+ String volume = "vol-" + RandomStringUtils.randomNumeric(5);
+ String bucket = "buc-" + RandomStringUtils.randomNumeric(5);
+ store.createVolume(volume);
+ OzoneVolume vol = store.getVolume(volume);
+ vol.createBucket(bucket);
+ OzoneBucket volbucket = vol.getBucket(bucket);
+
+ String key = "key-";
+ byte[] value = RandomStringUtils.randomAscii(10240).getBytes(UTF_8);
+ OzoneOutputStream oneKey = volbucket.createKey(
Review Comment:
Created a separated subroutine createKeys and referenced it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] jyotirmoy-gh commented on a diff in pull request #3896: HDDS-7427 - [snapshot] Add unit-testcases for Ozone snapshot feature.
Posted by GitBox <gi...@apache.org>.
jyotirmoy-gh commented on code in PR #3896:
URL: https://github.com/apache/ozone/pull/3896#discussion_r1006630471
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshot.java:
##########
@@ -383,6 +383,98 @@ public void checkKey() throws Exception {
snapshotKeyPrefix + key1);
}
+ @Test
+ public void testListDeleteKey()
+ throws IOException, InterruptedException, TimeoutException {
+ String volume = "vol-" + RandomStringUtils.randomNumeric(5);
+ String bucket = "buc-" + RandomStringUtils.randomNumeric(5);
+ store.createVolume(volume);
+ OzoneVolume vol = store.getVolume(volume);
+ vol.createBucket(bucket);
+ OzoneBucket volbucket = vol.getBucket(bucket);
+
+ String key = "key-";
+ byte[] value = RandomStringUtils.randomAscii(10240).getBytes(UTF_8);
+ OzoneOutputStream oneKey = volbucket.createKey(
+ key + RandomStringUtils.randomNumeric(5),
+ value.length, RATIS, ONE,
+ new HashMap<>());
+ oneKey.write(value);
+ oneKey.close();
+
+ String snapshotKeyPrefix = createSnapshot(volume, bucket);
+ Iterator<? extends OzoneKey> volBucketIter =
+ volbucket.listKeys(snapshotKeyPrefix + "key-");
+ int volBucketKeyCount = 0;
+ while (volBucketIter.hasNext()) {
+ volBucketIter.next();
+ volBucketKeyCount++;
+ }
+ Assert.assertEquals(1, volBucketKeyCount);
+
+ deleteKeys(volbucket);
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org