You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "swamirishi (via GitHub)" <gi...@apache.org> on 2025/03/12 01:21:08 UTC
[PR] HDDS-12562. Reclaimable Directory entry filter for reclaiming deleted directory entries [ozone]
swamirishi opened a new pull request, #8055:
URL: https://github.com/apache/ozone/pull/8055
## What changes were proposed in this pull request?
Currently deleted directory entries are reclaimed by just peeking into the previous snapshot without taking locks and the subdirectories and sub keys are never removed from the active om's directoryTable & fileTable. Whenever we are processing these deletes locks should be taken before deleting so that these don't work on inconsistent data.
As part of this patch we implement ReclaimableDirectoryFilter which extends abstract class ReclaimableFilter and check whether this directory entry corresponding to the object exists in the previous snapshot's directory table.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-12562
## How was this patch tested?
Unit test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12562. Reclaimable Directory entry filter for reclaiming deleted directory entries [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8055:
URL: https://github.com/apache/ozone/pull/8055#discussion_r2073989914
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/filter/ReclaimableDirFilter.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot.filter;
+
+import java.io.IOException;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.ozone.om.KeyManager;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.SnapshotChainManager;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.SnapshotInfo;
+import org.apache.hadoop.ozone.om.lock.IOzoneManagerLock;
+import org.apache.hadoop.ozone.om.snapshot.ReferenceCounted;
+
+/**
+ * Filter to return deleted directories which are reclaimable based on their presence in previous snapshot in
+ * the snapshot chain.
+ */
+public class ReclaimableDirFilter extends ReclaimableFilter<OmKeyInfo> {
+
+ /**
+ * Filter to return deleted directories which are reclaimable based on their presence in previous snapshot in
+ * the snapshot chain.
+ */
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12562. Reclaimable Directory entry filter for reclaiming deleted directory entries [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8055:
URL: https://github.com/apache/ozone/pull/8055#discussion_r2075217969
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/filter/ReclaimableDirFilter.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot.filter;
+
+import java.io.IOException;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.ozone.om.KeyManager;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.SnapshotChainManager;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.SnapshotInfo;
+import org.apache.hadoop.ozone.om.lock.IOzoneManagerLock;
+import org.apache.hadoop.ozone.om.snapshot.ReferenceCounted;
+
+/**
+ * Class to filter out deleted directories which are reclaimable based on their presence in previous snapshot in
+ * the snapshot chain.
+ */
+public class ReclaimableDirFilter extends ReclaimableFilter<OmKeyInfo> {
+
+ public ReclaimableDirFilter(OzoneManager ozoneManager,
+ OmSnapshotManager omSnapshotManager, SnapshotChainManager snapshotChainManager,
+ SnapshotInfo currentSnapshotInfo, KeyManager keyManager,
+ IOzoneManagerLock lock) {
+ super(ozoneManager, omSnapshotManager, snapshotChainManager, currentSnapshotInfo, keyManager, lock, 1);
+ }
+
+ @Override
+ protected String getVolumeName(Table.KeyValue<String, OmKeyInfo> keyValue) throws IOException {
+ return keyValue.getValue().getVolumeName();
+ }
+
+ @Override
+ protected String getBucketName(Table.KeyValue<String, OmKeyInfo> keyValue) throws IOException {
+ return keyValue.getValue().getBucketName();
+ }
+
+ @Override
+ protected Boolean isReclaimable(Table.KeyValue<String, OmKeyInfo> deletedDirInfo) throws IOException {
+ ReferenceCounted<OmSnapshot> previousSnapshot = getPreviousOmSnapshot(0);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12562. Reclaimable Directory entry filter for reclaiming deleted directory entries [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8055:
URL: https://github.com/apache/ozone/pull/8055#discussion_r2073990443
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/filter/ReclaimableDirFilter.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot.filter;
+
+import java.io.IOException;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.ozone.om.KeyManager;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.SnapshotChainManager;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.SnapshotInfo;
+import org.apache.hadoop.ozone.om.lock.IOzoneManagerLock;
+import org.apache.hadoop.ozone.om.snapshot.ReferenceCounted;
+
+/**
+ * Filter to return deleted directories which are reclaimable based on their presence in previous snapshot in
Review Comment:
done for reclaimable rename filter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12562. Reclaimable Directory entry filter for reclaiming deleted directory entries [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8055:
URL: https://github.com/apache/ozone/pull/8055#discussion_r2049311015
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java:
##########
@@ -731,6 +732,33 @@ public List<Table.KeyValue<String, String>> getRenamesKeyEntries(
}
}
+ @Override
+ public CheckedFunction<KeyManager, OmDirectoryInfo, IOException> getPreviousSnapshotOzoneDirInfo(
Review Comment:
This would be used in the DirectoryDeletingService where the deletedDeletedDirectory table would have OmKeyInfo whereas the directory table would have OmDirectoryInfo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12562. Reclaimable Directory entry filter for reclaiming deleted directory entries [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8055:
URL: https://github.com/apache/ozone/pull/8055#discussion_r2074680713
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/filter/ReclaimableDirFilter.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot.filter;
+
+import java.io.IOException;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.ozone.om.KeyManager;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.SnapshotChainManager;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.SnapshotInfo;
+import org.apache.hadoop.ozone.om.lock.IOzoneManagerLock;
+import org.apache.hadoop.ozone.om.snapshot.ReferenceCounted;
+
+/**
+ * Class to filter out deleted directories which are reclaimable based on their presence in previous snapshot in
+ * the snapshot chain.
+ */
+public class ReclaimableDirFilter extends ReclaimableFilter<OmKeyInfo> {
+
+ public ReclaimableDirFilter(OzoneManager ozoneManager,
+ OmSnapshotManager omSnapshotManager, SnapshotChainManager snapshotChainManager,
+ SnapshotInfo currentSnapshotInfo, KeyManager keyManager,
+ IOzoneManagerLock lock) {
+ super(ozoneManager, omSnapshotManager, snapshotChainManager, currentSnapshotInfo, keyManager, lock, 1);
+ }
+
+ @Override
+ protected String getVolumeName(Table.KeyValue<String, OmKeyInfo> keyValue) throws IOException {
+ return keyValue.getValue().getVolumeName();
+ }
+
+ @Override
+ protected String getBucketName(Table.KeyValue<String, OmKeyInfo> keyValue) throws IOException {
+ return keyValue.getValue().getBucketName();
+ }
+
+ @Override
+ protected Boolean isReclaimable(Table.KeyValue<String, OmKeyInfo> deletedDirInfo) throws IOException {
+ ReferenceCounted<OmSnapshot> previousSnapshot = getPreviousOmSnapshot(0);
+ KeyManager prevKeyManager = previousSnapshot == null ? null : previousSnapshot.get().getKeyManager();
+ return isDirReclaimable(getVolumeId(), getBucketInfo(), deletedDirInfo.getValue(), getKeyManager(), prevKeyManager);
+ }
+
+ private boolean isDirReclaimable(long volumeId, OmBucketInfo bucketInfo, OmKeyInfo dirInfo,
+ KeyManager keyManager, KeyManager previousKeyManager) throws IOException {
+ if (previousKeyManager == null) {
+ return true;
+ }
+ OmDirectoryInfo prevDirectoryInfo =
+ keyManager.getPreviousSnapshotOzoneDirInfo(volumeId, bucketInfo, dirInfo).apply(previousKeyManager);
+ return prevDirectoryInfo == null || prevDirectoryInfo.getObjectID() != dirInfo.getObjectID();
Review Comment:
yeah
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12562. Reclaimable Directory entry filter for reclaiming deleted directory entries [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8055:
URL: https://github.com/apache/ozone/pull/8055#discussion_r2075220310
##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/filter/TestReclaimableDirFilter.java:
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot.filter;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.ozone.om.KeyManager;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.SnapshotChainManager;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.SnapshotInfo;
+import org.apache.hadoop.ozone.om.lock.IOzoneManagerLock;
+import org.apache.hadoop.ozone.om.snapshot.ReferenceCounted;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.rocksdb.RocksDBException;
+
+/**
+ * Test class for ReclaimableDirFilter.
+ */
+public class TestReclaimableDirFilter extends AbstractReclaimableFilterTest {
+ @Override
+ protected ReclaimableFilter initializeFilter(OzoneManager om, OmSnapshotManager snapshotManager,
+ SnapshotChainManager chainManager, SnapshotInfo currentSnapshotInfo,
+ KeyManager km, IOzoneManagerLock lock,
+ int numberOfPreviousSnapshotsFromChain) {
+ return new ReclaimableDirFilter(om, snapshotManager, chainManager, currentSnapshotInfo, km, lock);
+ }
+
+ List<Arguments> testReclaimableFilterArguments() {
+ List<Arguments> arguments = new ArrayList<>();
+ for (int i = 0; i < 3; i++) {
+ for (int j = 0; j < 5; j++) {
+ arguments.add(Arguments.of(i, j));
+ }
+ }
+ return arguments;
+ }
+
+ private void testReclaimableDirFilter(String volume, String bucket, int index,
+ OmKeyInfo dirInfo, OmDirectoryInfo prevDirInfo,
+ Boolean expectedValue)
+ throws IOException {
+ List<SnapshotInfo> snapshotInfos = getLastSnapshotInfos(volume, bucket, 1, index);
+ SnapshotInfo prevSnapshotInfo = snapshotInfos.get(0);
+ OmBucketInfo bucketInfo = getOzoneManager().getBucketInfo(volume, bucket);
+ long volumeId = getOzoneManager().getMetadataManager().getVolumeId(volume);
+ KeyManager keyManager = getKeyManager();
+ if (prevSnapshotInfo != null) {
+ ReferenceCounted<OmSnapshot> prevSnap = Optional.ofNullable(prevSnapshotInfo)
+ .map(info -> {
+ try {
+ return getOmSnapshotManager().getActiveSnapshot(volume, bucket, info.getName());
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }).orElse(null);
+ mockOmSnapshot(prevSnap);
+ when(keyManager.getPreviousSnapshotOzoneDirInfo(eq(volumeId), eq(bucketInfo), eq(dirInfo)))
+ .thenReturn((km) -> prevDirInfo);
+ }
+
+ when(dirInfo.getVolumeName()).thenReturn(volume);
+ when(dirInfo.getBucketName()).thenReturn(bucket);
+ assertEquals(expectedValue, getReclaimableFilter().apply(Table.newKeyValue("key", dirInfo)));
+ }
+
+ private OmKeyInfo getMockedOmKeyInfo(long objectId) {
+ OmKeyInfo keyInfo = mock(OmKeyInfo.class);
+ when(keyInfo.getObjectID()).thenReturn(objectId);
+ return keyInfo;
+ }
+
+ private OmDirectoryInfo getMockedOmDirInfo(long objectId) {
+ OmDirectoryInfo keyInfo = mock(OmDirectoryInfo.class);
+ when(keyInfo.getObjectID()).thenReturn(objectId);
+ return keyInfo;
+ }
+
+ private KeyManager mockOmSnapshot(ReferenceCounted<OmSnapshot> snapshot) {
+ if (snapshot != null) {
+ OmSnapshot omSnapshot = snapshot.get();
+ KeyManager keyManager = mock(KeyManager.class);
+ when(omSnapshot.getKeyManager()).thenReturn(keyManager);
+ return keyManager;
+ }
+ return null;
+ }
+
+ @ParameterizedTest
+ @MethodSource("testReclaimableFilterArguments")
+ public void testNonReclaimableDirectory(int actualNumberOfSnapshots, int index) throws IOException, RocksDBException {
+ setup(1, actualNumberOfSnapshots, index, 4, 2);
+ String volume = getVolumes().get(3);
+ String bucket = getBuckets().get(1);
+ index = Math.min(index, actualNumberOfSnapshots);
+ OmKeyInfo dirInfo = getMockedOmKeyInfo(1);
+ OmDirectoryInfo prevDirectoryInfo = index - 1 >= 0 ? getMockedOmDirInfo(1) : null;
+ testReclaimableDirFilter(volume, bucket, index, dirInfo, prevDirectoryInfo, prevDirectoryInfo == null);
Review Comment:
yeah
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12562. Reclaimable Directory entry filter for reclaiming deleted directory entries [ozone]
Posted by "hemantk-12 (via GitHub)" <gi...@apache.org>.
hemantk-12 commented on code in PR #8055:
URL: https://github.com/apache/ozone/pull/8055#discussion_r2022226457
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManager.java:
##########
@@ -134,6 +136,18 @@ List<Table.KeyValue<String, String>> getRenamesKeyEntries(
String volume, String bucket, String startKey, int size) throws IOException;
+ /**
+ * Returns the previous snapshot's ozone keyInfo corresponding for the object.
Review Comment:
```suggestion
* Returns the previous snapshot's ozone DirectoryInfo corresponding for the object.
```
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManager.java:
##########
@@ -134,6 +136,18 @@ List<Table.KeyValue<String, String>> getRenamesKeyEntries(
String volume, String bucket, String startKey, int size) throws IOException;
+ /**
+ * Returns the previous snapshot's ozone keyInfo corresponding for the object.
+ */
+ CheckedFunction<KeyManager, OmDirectoryInfo, IOException> getPreviousSnapshotOzoneDirInfo(
+ long volumeId, OmBucketInfo bucketInfo, OmDirectoryInfo directoryInfo) throws IOException;
+
+ /**
+ * Returns the previous snapshot's ozone keyInfo corresponding for the object.
+ */
+ CheckedFunction<KeyManager, OmDirectoryInfo, IOException> getPreviousSnapshotOzoneDirInfo(
+ long volumeId, OmBucketInfo bucketInfo, OmKeyInfo directoryInfo) throws IOException;
Review Comment:
Is it supposed to be a `getPreviousSnapshotOzoneDirInfo` or `getPreviousSnapshotOzoneKeyInfo`? Same for `directoryInfo`.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/filter/ReclaimableDirFilter.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot.filter;
+
+import java.io.IOException;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.ozone.om.KeyManager;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.SnapshotChainManager;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.SnapshotInfo;
+import org.apache.hadoop.ozone.om.lock.IOzoneManagerLock;
+import org.apache.hadoop.ozone.om.snapshot.ReferenceCounted;
+
+/**
+ * Filter to return deleted directories which are reclaimable based on their presence in previous snapshot in
Review Comment:
nit: please reword the comment. It doesn't return deleted directories rather filters reclaimable dirs.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/filter/ReclaimableDirFilter.java:
##########
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot.filter;
+
+import java.io.IOException;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.ozone.om.KeyManager;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.SnapshotChainManager;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.SnapshotInfo;
+import org.apache.hadoop.ozone.om.lock.IOzoneManagerLock;
+import org.apache.hadoop.ozone.om.snapshot.ReferenceCounted;
+
+/**
+ * Filter to return deleted directories which are reclaimable based on their presence in previous snapshot in
+ * the snapshot chain.
+ */
+public class ReclaimableDirFilter extends ReclaimableFilter<OmKeyInfo> {
+
+ /**
+ * Filter to return deleted directories which are reclaimable based on their presence in previous snapshot in
+ * the snapshot chain.
+ */
Review Comment:
nit: redundant comment.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java:
##########
@@ -731,6 +732,33 @@ public List<Table.KeyValue<String, String>> getRenamesKeyEntries(
}
}
+ @Override
+ public CheckedFunction<KeyManager, OmDirectoryInfo, IOException> getPreviousSnapshotOzoneDirInfo(
Review Comment:
Where and how will this be used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12562. Reclaimable Directory entry filter for reclaiming deleted directory entries [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8055:
URL: https://github.com/apache/ozone/pull/8055#discussion_r2074262006
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestKeyManagerImpl.java:
##########
@@ -1585,6 +1589,99 @@ void testGetNotExistedPart() throws IOException {
assertEquals(0, locationList.size());
}
+ private <T> Table<String, T> getMockedTable(Map<String, T> map) throws IOException {
+ Table<String, T> table = mock(Table.class);
+ when(table.get(anyString())).thenAnswer(i -> map.get(i.getArgument(0)));
+ when(table.getIfExist(anyString())).thenAnswer(i -> map.get(i.getArgument(0)));
+ return table;
+ }
+
+ private OmKeyInfo getMockedOmKeyInfo(OmBucketInfo bucketInfo, long parentId, String key, long objectId) {
+ OmKeyInfo omKeyInfo = mock(OmKeyInfo.class);
+ if (bucketInfo.getBucketLayout().isFileSystemOptimized()) {
+ when(omKeyInfo.getFileName()).thenReturn(key);
+ } else {
+ when(omKeyInfo.getParentObjectID()).thenReturn(parentId);
+ when(omKeyInfo.getKeyName()).thenReturn(key);
+ }
+ when(omKeyInfo.getObjectID()).thenReturn(objectId);
+ return omKeyInfo;
+ }
+
+ private OmDirectoryInfo getMockedOmDirInfo(long parentId, String key, long objectId) {
+ OmDirectoryInfo omKeyInfo = mock(OmDirectoryInfo.class);
+ when(omKeyInfo.getName()).thenReturn(key);
+ when(omKeyInfo.getParentObjectID()).thenReturn(parentId);
Review Comment:
I don't think so. Why do you ask?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12562. Reclaimable Directory entry filter for reclaiming deleted directory entries [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8055:
URL: https://github.com/apache/ozone/pull/8055#discussion_r2075219047
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestKeyManagerImpl.java:
##########
@@ -1585,6 +1589,99 @@ void testGetNotExistedPart() throws IOException {
assertEquals(0, locationList.size());
}
+ private <T> Table<String, T> getMockedTable(Map<String, T> map) throws IOException {
+ Table<String, T> table = mock(Table.class);
+ when(table.get(anyString())).thenAnswer(i -> map.get(i.getArgument(0)));
+ when(table.getIfExist(anyString())).thenAnswer(i -> map.get(i.getArgument(0)));
+ return table;
+ }
+
+ private OmKeyInfo getMockedOmKeyInfo(OmBucketInfo bucketInfo, long parentId, String key, long objectId) {
+ OmKeyInfo omKeyInfo = mock(OmKeyInfo.class);
+ if (bucketInfo.getBucketLayout().isFileSystemOptimized()) {
+ when(omKeyInfo.getFileName()).thenReturn(key);
+ } else {
+ when(omKeyInfo.getParentObjectID()).thenReturn(parentId);
+ when(omKeyInfo.getKeyName()).thenReturn(key);
+ }
+ when(omKeyInfo.getObjectID()).thenReturn(objectId);
+ return omKeyInfo;
+ }
+
+ private OmDirectoryInfo getMockedOmDirInfo(long parentId, String key, long objectId) {
+ OmDirectoryInfo omKeyInfo = mock(OmDirectoryInfo.class);
+ when(omKeyInfo.getName()).thenReturn(key);
+ when(omKeyInfo.getParentObjectID()).thenReturn(parentId);
+ when(omKeyInfo.getParentObjectID()).thenReturn(0L);
+ when(omKeyInfo.getObjectID()).thenReturn(objectId);
+ return omKeyInfo;
+ }
+
+ private String getPath(long volumeId, OmBucketInfo bucketInfo, OmKeyInfo omKeyInfo) {
+ if (bucketInfo.getBucketLayout().isFileSystemOptimized()) {
+ return volumeId + "/" + bucketInfo.getObjectID() + "/" + omKeyInfo.getParentObjectID() + "/" +
+ omKeyInfo.getFileName();
+ } else {
+ return bucketInfo.getVolumeName() + "/" + bucketInfo.getBucketName() + "/" + omKeyInfo.getKeyName();
+ }
+ }
+
+ private String getPath(long volumeId, OmBucketInfo bucketInfo, OmDirectoryInfo omDirInfo) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12562. Reclaimable Directory entry filter for reclaiming deleted directory entries [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #8055:
URL: https://github.com/apache/ozone/pull/8055#issuecomment-2855436584
Merged. Thanks @swamirishi thanks @hemantk-12 !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12562. Reclaimable Directory entry filter for reclaiming deleted directory entries [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #8055:
URL: https://github.com/apache/ozone/pull/8055#discussion_r2074188156
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestKeyManagerImpl.java:
##########
@@ -1585,6 +1589,99 @@ void testGetNotExistedPart() throws IOException {
assertEquals(0, locationList.size());
}
+ private <T> Table<String, T> getMockedTable(Map<String, T> map) throws IOException {
+ Table<String, T> table = mock(Table.class);
+ when(table.get(anyString())).thenAnswer(i -> map.get(i.getArgument(0)));
+ when(table.getIfExist(anyString())).thenAnswer(i -> map.get(i.getArgument(0)));
+ return table;
+ }
+
+ private OmKeyInfo getMockedOmKeyInfo(OmBucketInfo bucketInfo, long parentId, String key, long objectId) {
+ OmKeyInfo omKeyInfo = mock(OmKeyInfo.class);
+ if (bucketInfo.getBucketLayout().isFileSystemOptimized()) {
+ when(omKeyInfo.getFileName()).thenReturn(key);
+ } else {
+ when(omKeyInfo.getParentObjectID()).thenReturn(parentId);
+ when(omKeyInfo.getKeyName()).thenReturn(key);
+ }
+ when(omKeyInfo.getObjectID()).thenReturn(objectId);
+ return omKeyInfo;
+ }
+
+ private OmDirectoryInfo getMockedOmDirInfo(long parentId, String key, long objectId) {
+ OmDirectoryInfo omKeyInfo = mock(OmDirectoryInfo.class);
+ when(omKeyInfo.getName()).thenReturn(key);
+ when(omKeyInfo.getParentObjectID()).thenReturn(parentId);
+ when(omKeyInfo.getParentObjectID()).thenReturn(0L);
+ when(omKeyInfo.getObjectID()).thenReturn(objectId);
+ return omKeyInfo;
+ }
+
+ private String getPath(long volumeId, OmBucketInfo bucketInfo, OmKeyInfo omKeyInfo) {
+ if (bucketInfo.getBucketLayout().isFileSystemOptimized()) {
+ return volumeId + "/" + bucketInfo.getObjectID() + "/" + omKeyInfo.getParentObjectID() + "/" +
+ omKeyInfo.getFileName();
+ } else {
+ return bucketInfo.getVolumeName() + "/" + bucketInfo.getBucketName() + "/" + omKeyInfo.getKeyName();
+ }
+ }
+
+ private String getPath(long volumeId, OmBucketInfo bucketInfo, OmDirectoryInfo omDirInfo) {
Review Comment:
IMO be renamed as "getDirectoryKey" or something else would be more readable.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/filter/ReclaimableDirFilter.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot.filter;
+
+import java.io.IOException;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.ozone.om.KeyManager;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.SnapshotChainManager;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.SnapshotInfo;
+import org.apache.hadoop.ozone.om.lock.IOzoneManagerLock;
+import org.apache.hadoop.ozone.om.snapshot.ReferenceCounted;
+
+/**
+ * Class to filter out deleted directories which are reclaimable based on their presence in previous snapshot in
+ * the snapshot chain.
+ */
+public class ReclaimableDirFilter extends ReclaimableFilter<OmKeyInfo> {
+
+ public ReclaimableDirFilter(OzoneManager ozoneManager,
+ OmSnapshotManager omSnapshotManager, SnapshotChainManager snapshotChainManager,
+ SnapshotInfo currentSnapshotInfo, KeyManager keyManager,
+ IOzoneManagerLock lock) {
+ super(ozoneManager, omSnapshotManager, snapshotChainManager, currentSnapshotInfo, keyManager, lock, 1);
+ }
+
+ @Override
+ protected String getVolumeName(Table.KeyValue<String, OmKeyInfo> keyValue) throws IOException {
+ return keyValue.getValue().getVolumeName();
+ }
+
+ @Override
+ protected String getBucketName(Table.KeyValue<String, OmKeyInfo> keyValue) throws IOException {
+ return keyValue.getValue().getBucketName();
+ }
+
+ @Override
+ protected Boolean isReclaimable(Table.KeyValue<String, OmKeyInfo> deletedDirInfo) throws IOException {
+ ReferenceCounted<OmSnapshot> previousSnapshot = getPreviousOmSnapshot(0);
+ KeyManager prevKeyManager = previousSnapshot == null ? null : previousSnapshot.get().getKeyManager();
+ return isDirReclaimable(getVolumeId(), getBucketInfo(), deletedDirInfo.getValue(), getKeyManager(), prevKeyManager);
+ }
+
+ private boolean isDirReclaimable(long volumeId, OmBucketInfo bucketInfo, OmKeyInfo dirInfo,
+ KeyManager keyManager, KeyManager previousKeyManager) throws IOException {
+ if (previousKeyManager == null) {
+ return true;
+ }
+ OmDirectoryInfo prevDirectoryInfo =
+ keyManager.getPreviousSnapshotOzoneDirInfo(volumeId, bucketInfo, dirInfo).apply(previousKeyManager);
+ return prevDirectoryInfo == null || prevDirectoryInfo.getObjectID() != dirInfo.getObjectID();
Review Comment:
if the directory object is not in the previous snapshot, or the object id is not the same (the directory was deleted and then a new one was created), then it is reclaimable.
##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/filter/TestReclaimableDirFilter.java:
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot.filter;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.ozone.om.KeyManager;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.SnapshotChainManager;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.SnapshotInfo;
+import org.apache.hadoop.ozone.om.lock.IOzoneManagerLock;
+import org.apache.hadoop.ozone.om.snapshot.ReferenceCounted;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.rocksdb.RocksDBException;
+
+/**
+ * Test class for ReclaimableDirFilter.
+ */
+public class TestReclaimableDirFilter extends AbstractReclaimableFilterTest {
+ @Override
+ protected ReclaimableFilter initializeFilter(OzoneManager om, OmSnapshotManager snapshotManager,
+ SnapshotChainManager chainManager, SnapshotInfo currentSnapshotInfo,
+ KeyManager km, IOzoneManagerLock lock,
+ int numberOfPreviousSnapshotsFromChain) {
+ return new ReclaimableDirFilter(om, snapshotManager, chainManager, currentSnapshotInfo, km, lock);
+ }
+
+ List<Arguments> testReclaimableFilterArguments() {
+ List<Arguments> arguments = new ArrayList<>();
+ for (int i = 0; i < 3; i++) {
+ for (int j = 0; j < 5; j++) {
+ arguments.add(Arguments.of(i, j));
+ }
+ }
+ return arguments;
+ }
+
+ private void testReclaimableDirFilter(String volume, String bucket, int index,
+ OmKeyInfo dirInfo, OmDirectoryInfo prevDirInfo,
+ Boolean expectedValue)
+ throws IOException {
+ List<SnapshotInfo> snapshotInfos = getLastSnapshotInfos(volume, bucket, 1, index);
+ SnapshotInfo prevSnapshotInfo = snapshotInfos.get(0);
+ OmBucketInfo bucketInfo = getOzoneManager().getBucketInfo(volume, bucket);
+ long volumeId = getOzoneManager().getMetadataManager().getVolumeId(volume);
+ KeyManager keyManager = getKeyManager();
+ if (prevSnapshotInfo != null) {
+ ReferenceCounted<OmSnapshot> prevSnap = Optional.ofNullable(prevSnapshotInfo)
+ .map(info -> {
+ try {
+ return getOmSnapshotManager().getActiveSnapshot(volume, bucket, info.getName());
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }).orElse(null);
+ mockOmSnapshot(prevSnap);
+ when(keyManager.getPreviousSnapshotOzoneDirInfo(eq(volumeId), eq(bucketInfo), eq(dirInfo)))
+ .thenReturn((km) -> prevDirInfo);
+ }
+
+ when(dirInfo.getVolumeName()).thenReturn(volume);
+ when(dirInfo.getBucketName()).thenReturn(bucket);
+ assertEquals(expectedValue, getReclaimableFilter().apply(Table.newKeyValue("key", dirInfo)));
+ }
+
+ private OmKeyInfo getMockedOmKeyInfo(long objectId) {
+ OmKeyInfo keyInfo = mock(OmKeyInfo.class);
+ when(keyInfo.getObjectID()).thenReturn(objectId);
+ return keyInfo;
+ }
+
+ private OmDirectoryInfo getMockedOmDirInfo(long objectId) {
+ OmDirectoryInfo keyInfo = mock(OmDirectoryInfo.class);
+ when(keyInfo.getObjectID()).thenReturn(objectId);
+ return keyInfo;
+ }
+
+ private KeyManager mockOmSnapshot(ReferenceCounted<OmSnapshot> snapshot) {
+ if (snapshot != null) {
+ OmSnapshot omSnapshot = snapshot.get();
+ KeyManager keyManager = mock(KeyManager.class);
+ when(omSnapshot.getKeyManager()).thenReturn(keyManager);
+ return keyManager;
+ }
+ return null;
+ }
+
+ @ParameterizedTest
+ @MethodSource("testReclaimableFilterArguments")
+ public void testNonReclaimableDirectory(int actualNumberOfSnapshots, int index) throws IOException, RocksDBException {
+ setup(1, actualNumberOfSnapshots, index, 4, 2);
+ String volume = getVolumes().get(3);
+ String bucket = getBuckets().get(1);
+ index = Math.min(index, actualNumberOfSnapshots);
+ OmKeyInfo dirInfo = getMockedOmKeyInfo(1);
+ OmDirectoryInfo prevDirectoryInfo = index - 1 >= 0 ? getMockedOmDirInfo(1) : null;
+ testReclaimableDirFilter(volume, bucket, index, dirInfo, prevDirectoryInfo, prevDirectoryInfo == null);
+ }
+
+ @ParameterizedTest
+ @MethodSource("testReclaimableFilterArguments")
+ public void testReclaimableKeyWithDifferentObjId(int actualNumberOfSnapshots, int index)
+ throws IOException, RocksDBException {
+ setup(1, actualNumberOfSnapshots, index, 4, 2);
+ String volume = getVolumes().get(3);
+ String bucket = getBuckets().get(1);
+ index = Math.min(index, actualNumberOfSnapshots);
+ OmKeyInfo dirInfo = getMockedOmKeyInfo(1);
+ OmDirectoryInfo prevDirectoryInfo = index - 1 >= 0 ? getMockedOmDirInfo(2) : null;
+ testReclaimableDirFilter(volume, bucket, index, dirInfo, prevDirectoryInfo, true);
Review Comment:
the case where the same directory object is in the previous snapshot.
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestKeyManagerImpl.java:
##########
@@ -1585,6 +1589,99 @@ void testGetNotExistedPart() throws IOException {
assertEquals(0, locationList.size());
}
+ private <T> Table<String, T> getMockedTable(Map<String, T> map) throws IOException {
+ Table<String, T> table = mock(Table.class);
+ when(table.get(anyString())).thenAnswer(i -> map.get(i.getArgument(0)));
+ when(table.getIfExist(anyString())).thenAnswer(i -> map.get(i.getArgument(0)));
+ return table;
+ }
+
+ private OmKeyInfo getMockedOmKeyInfo(OmBucketInfo bucketInfo, long parentId, String key, long objectId) {
+ OmKeyInfo omKeyInfo = mock(OmKeyInfo.class);
+ if (bucketInfo.getBucketLayout().isFileSystemOptimized()) {
+ when(omKeyInfo.getFileName()).thenReturn(key);
+ } else {
+ when(omKeyInfo.getParentObjectID()).thenReturn(parentId);
+ when(omKeyInfo.getKeyName()).thenReturn(key);
+ }
+ when(omKeyInfo.getObjectID()).thenReturn(objectId);
+ return omKeyInfo;
+ }
+
+ private OmDirectoryInfo getMockedOmDirInfo(long parentId, String key, long objectId) {
+ OmDirectoryInfo omKeyInfo = mock(OmDirectoryInfo.class);
+ when(omKeyInfo.getName()).thenReturn(key);
+ when(omKeyInfo.getParentObjectID()).thenReturn(parentId);
Review Comment:
is this line redundant?
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/filter/ReclaimableDirFilter.java:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot.filter;
+
+import java.io.IOException;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.ozone.om.KeyManager;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.SnapshotChainManager;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.SnapshotInfo;
+import org.apache.hadoop.ozone.om.lock.IOzoneManagerLock;
+import org.apache.hadoop.ozone.om.snapshot.ReferenceCounted;
+
+/**
+ * Class to filter out deleted directories which are reclaimable based on their presence in previous snapshot in
+ * the snapshot chain.
+ */
+public class ReclaimableDirFilter extends ReclaimableFilter<OmKeyInfo> {
+
+ public ReclaimableDirFilter(OzoneManager ozoneManager,
+ OmSnapshotManager omSnapshotManager, SnapshotChainManager snapshotChainManager,
+ SnapshotInfo currentSnapshotInfo, KeyManager keyManager,
+ IOzoneManagerLock lock) {
+ super(ozoneManager, omSnapshotManager, snapshotChainManager, currentSnapshotInfo, keyManager, lock, 1);
+ }
+
+ @Override
+ protected String getVolumeName(Table.KeyValue<String, OmKeyInfo> keyValue) throws IOException {
+ return keyValue.getValue().getVolumeName();
+ }
+
+ @Override
+ protected String getBucketName(Table.KeyValue<String, OmKeyInfo> keyValue) throws IOException {
+ return keyValue.getValue().getBucketName();
+ }
+
+ @Override
+ protected Boolean isReclaimable(Table.KeyValue<String, OmKeyInfo> deletedDirInfo) throws IOException {
+ ReferenceCounted<OmSnapshot> previousSnapshot = getPreviousOmSnapshot(0);
Review Comment:
ReclaimableFilter.initializePreviousSnapshotsFromChain() should have an assert that previousOmSnapshots is exactly numberOfPreviousSnapshotsFromChain snapshots.
##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/filter/TestReclaimableDirFilter.java:
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot.filter;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.ozone.om.KeyManager;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.SnapshotChainManager;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.SnapshotInfo;
+import org.apache.hadoop.ozone.om.lock.IOzoneManagerLock;
+import org.apache.hadoop.ozone.om.snapshot.ReferenceCounted;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.rocksdb.RocksDBException;
+
+/**
+ * Test class for ReclaimableDirFilter.
+ */
+public class TestReclaimableDirFilter extends AbstractReclaimableFilterTest {
+ @Override
+ protected ReclaimableFilter initializeFilter(OzoneManager om, OmSnapshotManager snapshotManager,
+ SnapshotChainManager chainManager, SnapshotInfo currentSnapshotInfo,
+ KeyManager km, IOzoneManagerLock lock,
+ int numberOfPreviousSnapshotsFromChain) {
+ return new ReclaimableDirFilter(om, snapshotManager, chainManager, currentSnapshotInfo, km, lock);
+ }
+
+ List<Arguments> testReclaimableFilterArguments() {
+ List<Arguments> arguments = new ArrayList<>();
+ for (int i = 0; i < 3; i++) {
+ for (int j = 0; j < 5; j++) {
+ arguments.add(Arguments.of(i, j));
+ }
+ }
+ return arguments;
+ }
+
+ private void testReclaimableDirFilter(String volume, String bucket, int index,
+ OmKeyInfo dirInfo, OmDirectoryInfo prevDirInfo,
+ Boolean expectedValue)
+ throws IOException {
+ List<SnapshotInfo> snapshotInfos = getLastSnapshotInfos(volume, bucket, 1, index);
+ SnapshotInfo prevSnapshotInfo = snapshotInfos.get(0);
+ OmBucketInfo bucketInfo = getOzoneManager().getBucketInfo(volume, bucket);
+ long volumeId = getOzoneManager().getMetadataManager().getVolumeId(volume);
+ KeyManager keyManager = getKeyManager();
+ if (prevSnapshotInfo != null) {
+ ReferenceCounted<OmSnapshot> prevSnap = Optional.ofNullable(prevSnapshotInfo)
+ .map(info -> {
+ try {
+ return getOmSnapshotManager().getActiveSnapshot(volume, bucket, info.getName());
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }).orElse(null);
+ mockOmSnapshot(prevSnap);
+ when(keyManager.getPreviousSnapshotOzoneDirInfo(eq(volumeId), eq(bucketInfo), eq(dirInfo)))
+ .thenReturn((km) -> prevDirInfo);
+ }
+
+ when(dirInfo.getVolumeName()).thenReturn(volume);
+ when(dirInfo.getBucketName()).thenReturn(bucket);
+ assertEquals(expectedValue, getReclaimableFilter().apply(Table.newKeyValue("key", dirInfo)));
+ }
+
+ private OmKeyInfo getMockedOmKeyInfo(long objectId) {
+ OmKeyInfo keyInfo = mock(OmKeyInfo.class);
+ when(keyInfo.getObjectID()).thenReturn(objectId);
+ return keyInfo;
+ }
+
+ private OmDirectoryInfo getMockedOmDirInfo(long objectId) {
+ OmDirectoryInfo keyInfo = mock(OmDirectoryInfo.class);
+ when(keyInfo.getObjectID()).thenReturn(objectId);
+ return keyInfo;
+ }
+
+ private KeyManager mockOmSnapshot(ReferenceCounted<OmSnapshot> snapshot) {
+ if (snapshot != null) {
+ OmSnapshot omSnapshot = snapshot.get();
+ KeyManager keyManager = mock(KeyManager.class);
+ when(omSnapshot.getKeyManager()).thenReturn(keyManager);
+ return keyManager;
+ }
+ return null;
+ }
+
+ @ParameterizedTest
+ @MethodSource("testReclaimableFilterArguments")
+ public void testNonReclaimableDirectory(int actualNumberOfSnapshots, int index) throws IOException, RocksDBException {
+ setup(1, actualNumberOfSnapshots, index, 4, 2);
+ String volume = getVolumes().get(3);
+ String bucket = getBuckets().get(1);
+ index = Math.min(index, actualNumberOfSnapshots);
+ OmKeyInfo dirInfo = getMockedOmKeyInfo(1);
+ OmDirectoryInfo prevDirectoryInfo = index - 1 >= 0 ? getMockedOmDirInfo(1) : null;
+ testReclaimableDirFilter(volume, bucket, index, dirInfo, prevDirectoryInfo, prevDirectoryInfo == null);
+ }
+
+ @ParameterizedTest
+ @MethodSource("testReclaimableFilterArguments")
+ public void testReclaimableKeyWithDifferentObjId(int actualNumberOfSnapshots, int index)
+ throws IOException, RocksDBException {
+ setup(1, actualNumberOfSnapshots, index, 4, 2);
+ String volume = getVolumes().get(3);
+ String bucket = getBuckets().get(1);
+ index = Math.min(index, actualNumberOfSnapshots);
+ OmKeyInfo dirInfo = getMockedOmKeyInfo(1);
+ OmDirectoryInfo prevDirectoryInfo = index - 1 >= 0 ? getMockedOmDirInfo(2) : null;
+ testReclaimableDirFilter(volume, bucket, index, dirInfo, prevDirectoryInfo, true);
Review Comment:
should we also check the case where a different directory object is in the previous snapshot? In which case it should be reclaimable.
##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/filter/TestReclaimableDirFilter.java:
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot.filter;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.ozone.om.KeyManager;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.SnapshotChainManager;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.SnapshotInfo;
+import org.apache.hadoop.ozone.om.lock.IOzoneManagerLock;
+import org.apache.hadoop.ozone.om.snapshot.ReferenceCounted;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.rocksdb.RocksDBException;
+
+/**
+ * Test class for ReclaimableDirFilter.
+ */
+public class TestReclaimableDirFilter extends AbstractReclaimableFilterTest {
+ @Override
+ protected ReclaimableFilter initializeFilter(OzoneManager om, OmSnapshotManager snapshotManager,
+ SnapshotChainManager chainManager, SnapshotInfo currentSnapshotInfo,
+ KeyManager km, IOzoneManagerLock lock,
+ int numberOfPreviousSnapshotsFromChain) {
+ return new ReclaimableDirFilter(om, snapshotManager, chainManager, currentSnapshotInfo, km, lock);
+ }
+
+ List<Arguments> testReclaimableFilterArguments() {
+ List<Arguments> arguments = new ArrayList<>();
+ for (int i = 0; i < 3; i++) {
+ for (int j = 0; j < 5; j++) {
+ arguments.add(Arguments.of(i, j));
+ }
+ }
+ return arguments;
+ }
+
+ private void testReclaimableDirFilter(String volume, String bucket, int index,
+ OmKeyInfo dirInfo, OmDirectoryInfo prevDirInfo,
+ Boolean expectedValue)
+ throws IOException {
+ List<SnapshotInfo> snapshotInfos = getLastSnapshotInfos(volume, bucket, 1, index);
+ SnapshotInfo prevSnapshotInfo = snapshotInfos.get(0);
+ OmBucketInfo bucketInfo = getOzoneManager().getBucketInfo(volume, bucket);
+ long volumeId = getOzoneManager().getMetadataManager().getVolumeId(volume);
+ KeyManager keyManager = getKeyManager();
+ if (prevSnapshotInfo != null) {
+ ReferenceCounted<OmSnapshot> prevSnap = Optional.ofNullable(prevSnapshotInfo)
+ .map(info -> {
+ try {
+ return getOmSnapshotManager().getActiveSnapshot(volume, bucket, info.getName());
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }).orElse(null);
+ mockOmSnapshot(prevSnap);
+ when(keyManager.getPreviousSnapshotOzoneDirInfo(eq(volumeId), eq(bucketInfo), eq(dirInfo)))
+ .thenReturn((km) -> prevDirInfo);
+ }
+
+ when(dirInfo.getVolumeName()).thenReturn(volume);
+ when(dirInfo.getBucketName()).thenReturn(bucket);
+ assertEquals(expectedValue, getReclaimableFilter().apply(Table.newKeyValue("key", dirInfo)));
+ }
+
+ private OmKeyInfo getMockedOmKeyInfo(long objectId) {
+ OmKeyInfo keyInfo = mock(OmKeyInfo.class);
+ when(keyInfo.getObjectID()).thenReturn(objectId);
+ return keyInfo;
+ }
+
+ private OmDirectoryInfo getMockedOmDirInfo(long objectId) {
+ OmDirectoryInfo keyInfo = mock(OmDirectoryInfo.class);
+ when(keyInfo.getObjectID()).thenReturn(objectId);
+ return keyInfo;
+ }
+
+ private KeyManager mockOmSnapshot(ReferenceCounted<OmSnapshot> snapshot) {
+ if (snapshot != null) {
+ OmSnapshot omSnapshot = snapshot.get();
+ KeyManager keyManager = mock(KeyManager.class);
+ when(omSnapshot.getKeyManager()).thenReturn(keyManager);
+ return keyManager;
+ }
+ return null;
+ }
+
+ @ParameterizedTest
+ @MethodSource("testReclaimableFilterArguments")
+ public void testNonReclaimableDirectory(int actualNumberOfSnapshots, int index) throws IOException, RocksDBException {
+ setup(1, actualNumberOfSnapshots, index, 4, 2);
+ String volume = getVolumes().get(3);
+ String bucket = getBuckets().get(1);
+ index = Math.min(index, actualNumberOfSnapshots);
+ OmKeyInfo dirInfo = getMockedOmKeyInfo(1);
+ OmDirectoryInfo prevDirectoryInfo = index - 1 >= 0 ? getMockedOmDirInfo(1) : null;
+ testReclaimableDirFilter(volume, bucket, index, dirInfo, prevDirectoryInfo, prevDirectoryInfo == null);
Review Comment:
the case where the directory is not in the previous snapshot.
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestKeyManagerImpl.java:
##########
@@ -1585,6 +1589,99 @@ void testGetNotExistedPart() throws IOException {
assertEquals(0, locationList.size());
}
+ private <T> Table<String, T> getMockedTable(Map<String, T> map) throws IOException {
+ Table<String, T> table = mock(Table.class);
+ when(table.get(anyString())).thenAnswer(i -> map.get(i.getArgument(0)));
+ when(table.getIfExist(anyString())).thenAnswer(i -> map.get(i.getArgument(0)));
+ return table;
+ }
+
+ private OmKeyInfo getMockedOmKeyInfo(OmBucketInfo bucketInfo, long parentId, String key, long objectId) {
+ OmKeyInfo omKeyInfo = mock(OmKeyInfo.class);
+ if (bucketInfo.getBucketLayout().isFileSystemOptimized()) {
+ when(omKeyInfo.getFileName()).thenReturn(key);
+ } else {
+ when(omKeyInfo.getParentObjectID()).thenReturn(parentId);
+ when(omKeyInfo.getKeyName()).thenReturn(key);
+ }
+ when(omKeyInfo.getObjectID()).thenReturn(objectId);
+ return omKeyInfo;
+ }
+
+ private OmDirectoryInfo getMockedOmDirInfo(long parentId, String key, long objectId) {
+ OmDirectoryInfo omKeyInfo = mock(OmDirectoryInfo.class);
+ when(omKeyInfo.getName()).thenReturn(key);
+ when(omKeyInfo.getParentObjectID()).thenReturn(parentId);
Review Comment:
this line will not take effect as it will be replaced by the next line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12562. Reclaimable Directory entry filter for reclaiming deleted directory entries [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8055:
URL: https://github.com/apache/ozone/pull/8055#discussion_r2074262006
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestKeyManagerImpl.java:
##########
@@ -1585,6 +1589,99 @@ void testGetNotExistedPart() throws IOException {
assertEquals(0, locationList.size());
}
+ private <T> Table<String, T> getMockedTable(Map<String, T> map) throws IOException {
+ Table<String, T> table = mock(Table.class);
+ when(table.get(anyString())).thenAnswer(i -> map.get(i.getArgument(0)));
+ when(table.getIfExist(anyString())).thenAnswer(i -> map.get(i.getArgument(0)));
+ return table;
+ }
+
+ private OmKeyInfo getMockedOmKeyInfo(OmBucketInfo bucketInfo, long parentId, String key, long objectId) {
+ OmKeyInfo omKeyInfo = mock(OmKeyInfo.class);
+ if (bucketInfo.getBucketLayout().isFileSystemOptimized()) {
+ when(omKeyInfo.getFileName()).thenReturn(key);
+ } else {
+ when(omKeyInfo.getParentObjectID()).thenReturn(parentId);
+ when(omKeyInfo.getKeyName()).thenReturn(key);
+ }
+ when(omKeyInfo.getObjectID()).thenReturn(objectId);
+ return omKeyInfo;
+ }
+
+ private OmDirectoryInfo getMockedOmDirInfo(long parentId, String key, long objectId) {
+ OmDirectoryInfo omKeyInfo = mock(OmDirectoryInfo.class);
+ when(omKeyInfo.getName()).thenReturn(key);
+ when(omKeyInfo.getParentObjectID()).thenReturn(parentId);
Review Comment:
I don't think so. Why do you ask?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12562. Reclaimable Directory entry filter for reclaiming deleted directory entries [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8055:
URL: https://github.com/apache/ozone/pull/8055#discussion_r2074681267
##########
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/snapshot/filter/TestReclaimableDirFilter.java:
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot.filter;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.ozone.om.KeyManager;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.OzoneManager;
+import org.apache.hadoop.ozone.om.SnapshotChainManager;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.SnapshotInfo;
+import org.apache.hadoop.ozone.om.lock.IOzoneManagerLock;
+import org.apache.hadoop.ozone.om.snapshot.ReferenceCounted;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.rocksdb.RocksDBException;
+
+/**
+ * Test class for ReclaimableDirFilter.
+ */
+public class TestReclaimableDirFilter extends AbstractReclaimableFilterTest {
+ @Override
+ protected ReclaimableFilter initializeFilter(OzoneManager om, OmSnapshotManager snapshotManager,
+ SnapshotChainManager chainManager, SnapshotInfo currentSnapshotInfo,
+ KeyManager km, IOzoneManagerLock lock,
+ int numberOfPreviousSnapshotsFromChain) {
+ return new ReclaimableDirFilter(om, snapshotManager, chainManager, currentSnapshotInfo, km, lock);
+ }
+
+ List<Arguments> testReclaimableFilterArguments() {
+ List<Arguments> arguments = new ArrayList<>();
+ for (int i = 0; i < 3; i++) {
+ for (int j = 0; j < 5; j++) {
+ arguments.add(Arguments.of(i, j));
+ }
+ }
+ return arguments;
+ }
+
+ private void testReclaimableDirFilter(String volume, String bucket, int index,
+ OmKeyInfo dirInfo, OmDirectoryInfo prevDirInfo,
+ Boolean expectedValue)
+ throws IOException {
+ List<SnapshotInfo> snapshotInfos = getLastSnapshotInfos(volume, bucket, 1, index);
+ SnapshotInfo prevSnapshotInfo = snapshotInfos.get(0);
+ OmBucketInfo bucketInfo = getOzoneManager().getBucketInfo(volume, bucket);
+ long volumeId = getOzoneManager().getMetadataManager().getVolumeId(volume);
+ KeyManager keyManager = getKeyManager();
+ if (prevSnapshotInfo != null) {
+ ReferenceCounted<OmSnapshot> prevSnap = Optional.ofNullable(prevSnapshotInfo)
+ .map(info -> {
+ try {
+ return getOmSnapshotManager().getActiveSnapshot(volume, bucket, info.getName());
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }).orElse(null);
+ mockOmSnapshot(prevSnap);
+ when(keyManager.getPreviousSnapshotOzoneDirInfo(eq(volumeId), eq(bucketInfo), eq(dirInfo)))
+ .thenReturn((km) -> prevDirInfo);
+ }
+
+ when(dirInfo.getVolumeName()).thenReturn(volume);
+ when(dirInfo.getBucketName()).thenReturn(bucket);
+ assertEquals(expectedValue, getReclaimableFilter().apply(Table.newKeyValue("key", dirInfo)));
+ }
+
+ private OmKeyInfo getMockedOmKeyInfo(long objectId) {
+ OmKeyInfo keyInfo = mock(OmKeyInfo.class);
+ when(keyInfo.getObjectID()).thenReturn(objectId);
+ return keyInfo;
+ }
+
+ private OmDirectoryInfo getMockedOmDirInfo(long objectId) {
+ OmDirectoryInfo keyInfo = mock(OmDirectoryInfo.class);
+ when(keyInfo.getObjectID()).thenReturn(objectId);
+ return keyInfo;
+ }
+
+ private KeyManager mockOmSnapshot(ReferenceCounted<OmSnapshot> snapshot) {
+ if (snapshot != null) {
+ OmSnapshot omSnapshot = snapshot.get();
+ KeyManager keyManager = mock(KeyManager.class);
+ when(omSnapshot.getKeyManager()).thenReturn(keyManager);
+ return keyManager;
+ }
+ return null;
+ }
+
+ @ParameterizedTest
+ @MethodSource("testReclaimableFilterArguments")
+ public void testNonReclaimableDirectory(int actualNumberOfSnapshots, int index) throws IOException, RocksDBException {
+ setup(1, actualNumberOfSnapshots, index, 4, 2);
+ String volume = getVolumes().get(3);
+ String bucket = getBuckets().get(1);
+ index = Math.min(index, actualNumberOfSnapshots);
+ OmKeyInfo dirInfo = getMockedOmKeyInfo(1);
+ OmDirectoryInfo prevDirectoryInfo = index - 1 >= 0 ? getMockedOmDirInfo(1) : null;
+ testReclaimableDirFilter(volume, bucket, index, dirInfo, prevDirectoryInfo, prevDirectoryInfo == null);
+ }
+
+ @ParameterizedTest
+ @MethodSource("testReclaimableFilterArguments")
+ public void testReclaimableKeyWithDifferentObjId(int actualNumberOfSnapshots, int index)
+ throws IOException, RocksDBException {
+ setup(1, actualNumberOfSnapshots, index, 4, 2);
+ String volume = getVolumes().get(3);
+ String bucket = getBuckets().get(1);
+ index = Math.min(index, actualNumberOfSnapshots);
+ OmKeyInfo dirInfo = getMockedOmKeyInfo(1);
+ OmDirectoryInfo prevDirectoryInfo = index - 1 >= 0 ? getMockedOmDirInfo(2) : null;
+ testReclaimableDirFilter(volume, bucket, index, dirInfo, prevDirectoryInfo, true);
Review Comment:
yeah test case is already there.
testReclaimableKeyWithDifferentObjId
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12562. Reclaimable Directory entry filter for reclaiming deleted directory entries [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8055:
URL: https://github.com/apache/ozone/pull/8055#discussion_r2074670517
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestKeyManagerImpl.java:
##########
@@ -1585,6 +1589,99 @@ void testGetNotExistedPart() throws IOException {
assertEquals(0, locationList.size());
}
+ private <T> Table<String, T> getMockedTable(Map<String, T> map) throws IOException {
+ Table<String, T> table = mock(Table.class);
+ when(table.get(anyString())).thenAnswer(i -> map.get(i.getArgument(0)));
+ when(table.getIfExist(anyString())).thenAnswer(i -> map.get(i.getArgument(0)));
+ return table;
+ }
+
+ private OmKeyInfo getMockedOmKeyInfo(OmBucketInfo bucketInfo, long parentId, String key, long objectId) {
+ OmKeyInfo omKeyInfo = mock(OmKeyInfo.class);
+ if (bucketInfo.getBucketLayout().isFileSystemOptimized()) {
+ when(omKeyInfo.getFileName()).thenReturn(key);
+ } else {
+ when(omKeyInfo.getParentObjectID()).thenReturn(parentId);
+ when(omKeyInfo.getKeyName()).thenReturn(key);
+ }
+ when(omKeyInfo.getObjectID()).thenReturn(objectId);
+ return omKeyInfo;
+ }
+
+ private OmDirectoryInfo getMockedOmDirInfo(long parentId, String key, long objectId) {
+ OmDirectoryInfo omKeyInfo = mock(OmDirectoryInfo.class);
+ when(omKeyInfo.getName()).thenReturn(key);
+ when(omKeyInfo.getParentObjectID()).thenReturn(parentId);
Review Comment:
yeah this was a mistake fixed it.
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestKeyManagerImpl.java:
##########
@@ -1585,6 +1589,99 @@ void testGetNotExistedPart() throws IOException {
assertEquals(0, locationList.size());
}
+ private <T> Table<String, T> getMockedTable(Map<String, T> map) throws IOException {
+ Table<String, T> table = mock(Table.class);
+ when(table.get(anyString())).thenAnswer(i -> map.get(i.getArgument(0)));
+ when(table.getIfExist(anyString())).thenAnswer(i -> map.get(i.getArgument(0)));
+ return table;
+ }
+
+ private OmKeyInfo getMockedOmKeyInfo(OmBucketInfo bucketInfo, long parentId, String key, long objectId) {
+ OmKeyInfo omKeyInfo = mock(OmKeyInfo.class);
+ if (bucketInfo.getBucketLayout().isFileSystemOptimized()) {
+ when(omKeyInfo.getFileName()).thenReturn(key);
+ } else {
+ when(omKeyInfo.getParentObjectID()).thenReturn(parentId);
+ when(omKeyInfo.getKeyName()).thenReturn(key);
+ }
+ when(omKeyInfo.getObjectID()).thenReturn(objectId);
+ return omKeyInfo;
+ }
+
+ private OmDirectoryInfo getMockedOmDirInfo(long parentId, String key, long objectId) {
+ OmDirectoryInfo omKeyInfo = mock(OmDirectoryInfo.class);
+ when(omKeyInfo.getName()).thenReturn(key);
+ when(omKeyInfo.getParentObjectID()).thenReturn(parentId);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12562. Reclaimable Directory entry filter for reclaiming deleted directory entries [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8055:
URL: https://github.com/apache/ozone/pull/8055#discussion_r2073995771
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManager.java:
##########
@@ -134,6 +136,18 @@ List<Table.KeyValue<String, String>> getRenamesKeyEntries(
String volume, String bucket, String startKey, int size) throws IOException;
+ /**
+ * Returns the previous snapshot's ozone keyInfo corresponding for the object.
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-12562. Reclaimable Directory entry filter for reclaiming deleted directory entries [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang merged PR #8055:
URL: https://github.com/apache/ozone/pull/8055
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org