You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2022/10/25 19:03:58 UTC
[GitHub] [ozone] nandakumar131 opened a new pull request, #3885: HDDS-7410. Ozone snapshot diff skeleton code.
nandakumar131 opened a new pull request, #3885:
URL: https://github.com/apache/ozone/pull/3885
Ozone snapshot diff skeleton code
HDDS-7410
## How was this patch tested?
Unit test to cover all the scenarios will be added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] hemantk-12 commented on a diff in pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
hemantk-12 commented on code in PR #3885:
URL: https://github.com/apache/ozone/pull/3885#discussion_r1008589835
##########
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdb/util/ManagedSSTFileReader.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ozone.rocksdb.util;
+
+import org.rocksdb.Options;
+import org.rocksdb.ReadOptions;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.SstFileReader;
+import org.rocksdb.SstFileReaderIterator;
+
+import java.io.Closeable;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.NoSuchElementException;
+import java.util.Spliterator;
+import java.util.Spliterators;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+public class ManagedSSTFileReader {
+
+ private final Collection<String> sstFiles;
+
+ public ManagedSSTFileReader(final Collection<String> sstFiles) {
+ this.sstFiles = sstFiles;
+ }
+ public Stream<String> getKeyStream() throws RocksDBException {
+ final OMSSTFileIterator iterator = new OMSSTFileIterator(sstFiles);
+ final Spliterator<String> spliterator = Spliterators
+ .spliteratorUnknownSize(iterator, 0);
+ return StreamSupport.stream(spliterator, false).onClose(iterator::close);
+ }
+
+ public interface ClosableIterator<T> extends Iterator<T>, Closeable {}
Review Comment:
What's the purpose of this interface? Why can't `OMSSTFileIterator` directly implement `Iterator` and `Closeable`?
##########
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdb/util/ManagedSSTFileReader.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ozone.rocksdb.util;
+
+import org.rocksdb.Options;
+import org.rocksdb.ReadOptions;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.SstFileReader;
+import org.rocksdb.SstFileReaderIterator;
+
+import java.io.Closeable;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.NoSuchElementException;
+import java.util.Spliterator;
+import java.util.Spliterators;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+public class ManagedSSTFileReader {
Review Comment:
nit/suggestion: I think `PascalCase` should be used for acronyms otherwise it becomes difficult to read when multiple acronyms put together. For example, `OMSSTFileIterator` should be `OmSstFileIterator` and `ManagedSSTFileReader` should be `ManagedSstFileReader`. Also in parity of existing stuff `OmSnapshot` and `OmSnapshotManager`.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot;
+
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.snapshot.SnapshotDiffReport.DiffType;
+import org.apache.hadoop.ozone.om.snapshot.SnapshotDiffReport.DiffReportEntry;
+
+import org.apache.ozone.rocksdb.util.ManagedSSTFileReader;
+import org.rocksdb.RocksDBException;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+
+public class SnapshotDiffManager {
+
+ public SnapshotDiffReport getSnapshotDiffReport(final String volume,
+ final String bucket,
+ final OmSnapshot fromSnapshot,
+ final OmSnapshot toSnapshot)
+ throws IOException, RocksDBException {
+
+ // TODO: Once RocksDBCheckpointDiffer exposes method to get list
+ // of delta SST files, plug it in here.
+
+ final Set<String> deltaFiles = Collections.emptySet();
+
+ // TODO: Filter out the files.
+
+ final Stream<String> keysToCheck = new ManagedSSTFileReader(deltaFiles)
+ .getKeyStream();
+
+ final BucketLayout bucketLayout = getBucketLayout(volume, bucket,
+ fromSnapshot.getMetadataManager());
+
+ final Table<String, OmKeyInfo> fsKeyTable = fromSnapshot
+ .getMetadataManager().getKeyTable(bucketLayout);
+ final Table<String, OmKeyInfo> tsKeyTable = toSnapshot
+ .getMetadataManager().getKeyTable(bucketLayout);
+
+ /*
+ * The reason for having ObjectID to KeyName mapping instead of OmKeyInfo
+ * is to reduce the memory footprint.
+ */
+ final Map<Long, String> oldObjIdToKeyMap = new HashMap<>();
+ // Long --> const. length
+ // String --> var. length "/dir1/dir2/dir3/dir4/dir5/key1"
+ final Map<Long, String> newObjIdToKeyMap = new HashMap<>();
+
+ final Set<Long> objectIDsToCheck = new HashSet<>();
+
+ keysToCheck.forEach(key -> {
+ try {
+ final OmKeyInfo oldKey = fsKeyTable.get(key);
+ final OmKeyInfo newKey = tsKeyTable.get(key);
+ if (areKeysEqual(oldKey, newKey)) {
+ // We don't have to do anything.
+ return;
+ }
+ if (oldKey != null) {
+ final long oldObjId = oldKey.getObjectID();
+ oldObjIdToKeyMap.put(oldObjId, oldKey.getKeyName());
+ objectIDsToCheck.add(oldObjId);
+ }
+ if (newKey != null) {
+ final long newObjId = newKey.getObjectID();
+ newObjIdToKeyMap.put(newObjId, newKey.getKeyName());
+ objectIDsToCheck.add(newObjId);
+ }
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ keysToCheck.close();
+
+ return new SnapshotDiffReport(volume, bucket, fromSnapshot.getName(),
+ toSnapshot.getName(), generateDiffReport(objectIDsToCheck,
+ oldObjIdToKeyMap, newObjIdToKeyMap));
+ }
+
+ private List<DiffReportEntry> generateDiffReport(
+ final Set<Long> objectIDsToCheck,
+ final Map<Long, String> oldObjIdToKeyMap,
+ final Map<Long, String> newObjIdToKeyMap) {
+
+ final List<DiffReportEntry> deleteDiffs = new ArrayList<>();
+ final List<DiffReportEntry> renameDiffs = new ArrayList<>();
+ final List<DiffReportEntry> createDiffs = new ArrayList<>();
+ final List<DiffReportEntry> modifyDiffs = new ArrayList<>();
+
+
+ for (Long id : objectIDsToCheck) {
+ /*
+ * This key can be
+ * -> Created after the old snapshot was taken, which means it will be
+ * missing in oldKeyTable and present in newKeyTable.
+ * -> Deleted after the old snapshot was taken, which means it will be
+ * present in oldKeyTable and missing in newKeyTable.
+ * -> Modified after the old snapshot was taken, which means it will be
+ * present in oldKeyTable and present in newKeyTable with same
+ * Object ID but with different metadata.
+ * -> Renamed after the old snapshot was taken, which means it will be
+ * present in oldKeyTable and present in newKeyTable but with different
+ * name and same Object ID.
+ */
+
+ final String oldKeyName = oldObjIdToKeyMap.get(id);
+ final String newKeyName = newObjIdToKeyMap.get(id);
+
+ if (oldKeyName == null && newKeyName == null) {
+ // This cannot happen.
+ continue;
+ }
+
+ // Key Created.
+ if (oldKeyName == null) {
+ createDiffs.add(DiffReportEntry.of(DiffType.CREATE, newKeyName));
+ continue;
+ }
+
+ // Key Deleted.
+ if(newKeyName == null) {
+ deleteDiffs.add(DiffReportEntry.of(DiffType.DELETE, oldKeyName));
+ continue;
+ }
+
+ // Key modified.
+ if(oldKeyName.equals(newKeyName)) {
+ modifyDiffs.add(DiffReportEntry.of(DiffType.MODIFY, newKeyName));
+ continue;
+ }
+
+ // Key Renamed.
+ renameDiffs.add(DiffReportEntry.of(DiffType.RENAME,
+ oldKeyName, newKeyName));
+ }
+ /*
+ * The order in which snap-diff should be applied
+ *
+ * 1. Delete diffs
+ * 2. Rename diffs
+ * 3. Create diffs
+ * 4. Modified diffs
+ *
+ * Consider the following scenario
+ *
+ * 1. File "A" is created.
+ * 2. File "B" is created.
+ * 3. File "C" is created.
+ * Snapshot "1" is taken.
+ *
+ * Case 1:
+ * 1. File "A" is deleted.
+ * 2. File "B" is renamed to "A".
+ * Snapshot "2" is taken.
+ *
+ * Snapshot diff should be applied in the following order:
+ * 1. Delete "A"
+ * 2. Rename "B" to "A"
+ *
+ *
+ * Case 2:
+ * 1. File "B" is renamed to "C".
+ * 2. File "B" is created.
+ * Snapshot "2" is taken.
+ *
+ * Snapshot diff should be applied in the following order:
+ * 1. Rename "B" to "C"
+ * 2. Create "B"
+ *
+ */
+
+ final List<DiffReportEntry> snapshotDiffs = new ArrayList<>();
+ snapshotDiffs.addAll(deleteDiffs);
+ snapshotDiffs.addAll(renameDiffs);
+ snapshotDiffs.addAll(createDiffs);
+ snapshotDiffs.addAll(modifyDiffs);
+ return snapshotDiffs;
+ }
+
+ private BucketLayout getBucketLayout(final String volume,
+ final String bucket,
+ final OMMetadataManager mManager)
+ throws IOException {
+ final String bucketTableKey = mManager.getBucketKey(volume, bucket);
+ return mManager.getBucketTable().get(bucketTableKey).getBucketLayout();
Review Comment:
Are `mManager.getBucketKey(volume, bucket)` and `mManager.getBucketTable().get(bucketTableKey)` nullable?
##########
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdb/util/ManagedSSTFileReader.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ozone.rocksdb.util;
+
+import org.rocksdb.Options;
+import org.rocksdb.ReadOptions;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.SstFileReader;
+import org.rocksdb.SstFileReaderIterator;
+
+import java.io.Closeable;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.NoSuchElementException;
+import java.util.Spliterator;
+import java.util.Spliterators;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+public class ManagedSSTFileReader {
+
+ private final Collection<String> sstFiles;
+
+ public ManagedSSTFileReader(final Collection<String> sstFiles) {
+ this.sstFiles = sstFiles;
+ }
+ public Stream<String> getKeyStream() throws RocksDBException {
+ final OMSSTFileIterator iterator = new OMSSTFileIterator(sstFiles);
+ final Spliterator<String> spliterator = Spliterators
+ .spliteratorUnknownSize(iterator, 0);
+ return StreamSupport.stream(spliterator, false).onClose(iterator::close);
+ }
+
+ public interface ClosableIterator<T> extends Iterator<T>, Closeable {}
+
+ private static class OMSSTFileIterator implements ClosableIterator<String> {
+
+ private final Iterator<String> fileNameIterator;
+ private final Options options;
+ private final ReadOptions readOptions;
+ private String currentFile;
+ private SstFileReader currentFileReader;
+ private SstFileReaderIterator currentFileIterator;
+
+ private OMSSTFileIterator(Collection<String> files) throws RocksDBException {
+ // TODO: Check if default Options and ReadOptions is enough.
+ this.options = new Options();
+ this.readOptions = new ReadOptions();
+ this.fileNameIterator = files.iterator();
+ moveToNextFile();
+ }
+
+ @Override
+ public boolean hasNext() {
+ try {
+ do {
+ if (currentFileIterator.isValid()) {
+ return true;
+ }
+ } while (moveToNextFile());
+ } catch (RocksDBException e) {
+ return false;
Review Comment:
Is it fine to swallow exception here? I won't feel safe if it is some kind of retryable exception.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -196,4 +203,40 @@ public static boolean isSnapshotKey(String[] keyParts) {
return (keyParts.length > 1) &&
(keyParts[0].compareTo(OM_SNAPSHOT_INDICATOR) == 0);
}
-}
+
+ public SnapshotDiffReport getSnapshotDiffReport(final String volume,
+ final String bucket,
+ final String fromSnapshot,
+ final String toSnapshot)
+ throws IOException {
+ // Validate fromSnapshot and toSnapshot
+ final SnapshotInfo fsInfo = getSnapshotInfo(volume, bucket, fromSnapshot);
+ final SnapshotInfo tsInfo = getSnapshotInfo(volume, bucket, toSnapshot);
+ verifySnapshotInfoForSnapDiff(fsInfo, tsInfo);
+
+ final String fsKey = SnapshotInfo.getTableKey(volume, bucket, fromSnapshot);
+ final String tsKey = SnapshotInfo.getTableKey(volume, bucket, toSnapshot);
+ try {
+ final OmSnapshot fs = snapshotCache.get(fsKey);
+ final OmSnapshot ts = snapshotCache.get(tsKey);
+ return snapshotDiffManager.getSnapshotDiffReport(volume, bucket, fs, ts);
+ } catch (ExecutionException | RocksDBException e) {
+ throw new IOException(e.getCause());
+ }
+ }
+
+ private void verifySnapshotInfoForSnapDiff(final SnapshotInfo fromSnapshot,
+ final SnapshotInfo toSnapshot)
+ throws IOException {
+ if ((fromSnapshot.getSnapshotStatus() != SnapshotStatus.SNAPSHOT_ACTIVE) ||
+ (toSnapshot.getSnapshotStatus() != SnapshotStatus.SNAPSHOT_ACTIVE)) {
+ // TODO: throw custom snapshot exception.
+ throw new IOException("Cannot generate snapshot diff for non-active " +
+ "snapshots.");
+ }
+ if (fromSnapshot.getCreationTime() > toSnapshot.getCreationTime()) {
+ throw new IOException("fromSnapshot should be older than to toSnapshot");
Review Comment:
Add actual snapshot names in exception message if available.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -196,4 +203,40 @@ public static boolean isSnapshotKey(String[] keyParts) {
return (keyParts.length > 1) &&
(keyParts[0].compareTo(OM_SNAPSHOT_INDICATOR) == 0);
}
-}
+
+ public SnapshotDiffReport getSnapshotDiffReport(final String volume,
+ final String bucket,
+ final String fromSnapshot,
+ final String toSnapshot)
+ throws IOException {
+ // Validate fromSnapshot and toSnapshot
+ final SnapshotInfo fsInfo = getSnapshotInfo(volume, bucket, fromSnapshot);
+ final SnapshotInfo tsInfo = getSnapshotInfo(volume, bucket, toSnapshot);
+ verifySnapshotInfoForSnapDiff(fsInfo, tsInfo);
+
+ final String fsKey = SnapshotInfo.getTableKey(volume, bucket, fromSnapshot);
+ final String tsKey = SnapshotInfo.getTableKey(volume, bucket, toSnapshot);
+ try {
+ final OmSnapshot fs = snapshotCache.get(fsKey);
+ final OmSnapshot ts = snapshotCache.get(tsKey);
+ return snapshotDiffManager.getSnapshotDiffReport(volume, bucket, fs, ts);
+ } catch (ExecutionException | RocksDBException e) {
+ throw new IOException(e.getCause());
+ }
+ }
+
+ private void verifySnapshotInfoForSnapDiff(final SnapshotInfo fromSnapshot,
+ final SnapshotInfo toSnapshot)
+ throws IOException {
+ if ((fromSnapshot.getSnapshotStatus() != SnapshotStatus.SNAPSHOT_ACTIVE) ||
+ (toSnapshot.getSnapshotStatus() != SnapshotStatus.SNAPSHOT_ACTIVE)) {
+ // TODO: throw custom snapshot exception.
+ throw new IOException("Cannot generate snapshot diff for non-active " +
+ "snapshots.");
+ }
+ if (fromSnapshot.getCreationTime() > toSnapshot.getCreationTime()) {
+ throw new IOException("fromSnapshot should be older than to toSnapshot");
+ }
+ }
+
+ }
Review Comment:
nit: alignment is bit off.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffReport.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot;
+
+import java.util.Collections;
+import java.util.List;
+
+public class SnapshotDiffReport {
+
+ private final static String LINE_SEPARATOR = System.getProperty(
+ "line.separator", "\n");
+
+ /**
+ * Types of the difference, which include CREATE, MODIFY, DELETE, and RENAME.
+ * Each type has a label for representation:
+ * + CREATE
+ * M MODIFY
+ * - DELETE
+ * R RENAME
+ */
+ public enum DiffType {
+ CREATE("+"),
+ MODIFY("M"),
+ DELETE("-"),
+ RENAME("R");
+
+ private final String label;
+
+ DiffType(String label) {
+ this.label = label;
+ }
+
+ public String getLabel() {
+ return label;
+ }
+ }
+
+ public static class DiffReportEntry {
+
+ /**
+ * The type of diff.
+ */
+ private final DiffType type;
+
+ /**
+ * Source File/Object path.
+ */
+ private final String sourcePath;
+
+ /**
+ * Destination File/Object path, if this is a re-name operation.
+ */
+ private final String targetPath;
+
+ private DiffReportEntry(final DiffType type, final String sourcePath,
+ final String targetPath) {
Review Comment:
nit: alignment is off.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffReport.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot;
+
+import java.util.Collections;
+import java.util.List;
+
+public class SnapshotDiffReport {
+
+ private final static String LINE_SEPARATOR = System.getProperty(
+ "line.separator", "\n");
+
+ /**
+ * Types of the difference, which include CREATE, MODIFY, DELETE, and RENAME.
+ * Each type has a label for representation:
+ * + CREATE
+ * M MODIFY
+ * - DELETE
+ * R RENAME
+ */
+ public enum DiffType {
+ CREATE("+"),
+ MODIFY("M"),
+ DELETE("-"),
+ RENAME("R");
+
+ private final String label;
+
+ DiffType(String label) {
+ this.label = label;
+ }
+
+ public String getLabel() {
+ return label;
+ }
+ }
+
+ public static class DiffReportEntry {
+
+ /**
+ * The type of diff.
+ */
+ private final DiffType type;
+
+ /**
+ * Source File/Object path.
+ */
+ private final String sourcePath;
+
+ /**
+ * Destination File/Object path, if this is a re-name operation.
+ */
+ private final String targetPath;
+
+ private DiffReportEntry(final DiffType type, final String sourcePath,
+ final String targetPath) {
+ this.type = type;
+ this.sourcePath = sourcePath;
+ this.targetPath = targetPath;
+ }
+
+ public static DiffReportEntry of(final DiffType type,
+ final String sourcePath) {
+ return of(type, sourcePath, null);
+ }
+
+ public static DiffReportEntry of(final DiffType type,
+ final String sourcePath,
+ final String targetPath) {
+ return new DiffReportEntry(type, sourcePath, null);
Review Comment:
`targetPath` instead `null`.
`DiffReportEntry(type, sourcePath, targetPath);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] nandakumar131 commented on a diff in pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
nandakumar131 commented on code in PR #3885:
URL: https://github.com/apache/ozone/pull/3885#discussion_r1022481146
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java:
##########
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot;
+
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.snapshot.SnapshotDiffReport.DiffType;
+import org.apache.hadoop.ozone.om.snapshot.SnapshotDiffReport.DiffReportEntry;
+
+import org.apache.ozone.rocksdb.util.ManagedSstFileReader;
+import org.apache.ozone.rocksdb.util.RdbUtil;
+import org.rocksdb.RocksDBException;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * Class to generate snapshot diff.
+ */
+public class SnapshotDiffManager {
+
+ public SnapshotDiffReport getSnapshotDiffReport(final String volume,
+ final String bucket,
+ final OmSnapshot fromSnapshot,
+ final OmSnapshot toSnapshot)
+ throws IOException, RocksDBException {
+
+ // TODO: Once RocksDBCheckpointDiffer exposes method to get list
+ // of delta SST files, plug it in here.
+
+ Set<String> fromSnapshotFiles = RdbUtil.getKeyTableSSTFiles(fromSnapshot
+ .getMetadataManager().getStore().getDbLocation().getPath());
+ Set<String> toSnapshotFiles = RdbUtil.getKeyTableSSTFiles(toSnapshot
+ .getMetadataManager().getStore().getDbLocation().getPath());
+
+ final Set<String> deltaFiles = new HashSet<>();
+ deltaFiles.addAll(fromSnapshotFiles);
+ deltaFiles.addAll(toSnapshotFiles);
+
+ // TODO: Filter out the files.
+
+ final Stream<String> keysToCheck = new ManagedSstFileReader(deltaFiles)
+ .getKeyStream();
+
+ final BucketLayout bucketLayout = getBucketLayout(volume, bucket,
+ fromSnapshot.getMetadataManager());
+
+ final Table<String, OmKeyInfo> fsKeyTable = fromSnapshot
+ .getMetadataManager().getKeyTable(bucketLayout);
+ final Table<String, OmKeyInfo> tsKeyTable = toSnapshot
+ .getMetadataManager().getKeyTable(bucketLayout);
+
+ /*
+ * The reason for having ObjectID to KeyName mapping instead of OmKeyInfo
+ * is to reduce the memory footprint.
+ */
+ final Map<Long, String> oldObjIdToKeyMap = new HashMap<>();
+ // Long --> const. length
+ // String --> var. length "/dir1/dir2/dir3/dir4/dir5/key1"
+ final Map<Long, String> newObjIdToKeyMap = new HashMap<>();
+
Review Comment:
Thansk @GeorgeJahad for taking a look at the change.
The HashMaps only stores the keys which are affected between snapshots (created, deleted, renamed)
For create, we only add it to newObjIdToKeyMap
For delete, we only add it to newObjIdToKeyMap
For rename, we add it to both newObjIdToKeyMap and newObjIdToKeyMap
We don't store the whole key metadata here in these HashMaps, we only store the object ID and key name.
Object ID is a long value which is of size 64-bit.
We will reach 100 bytes for an entry only if all the key paths are of length ~99 characters.
So we will use 1 gig of memory per HashMap only if we have 10 million renames with an average key size of ~99 characters.
In this case, our SnapshotDiff report will also be around 2 gig.
A single RPC call cannot handle 2 gig of data, so SnapshotDiff for large delta will fail.
We don't have paging logic built into SnapshotDiff yet. We need support for partial SnapshotDiff generation and paging logic at the client side to support large SnapshotDiff.
For now, we can add a check while we generate the SnapshotDiff and throw an error if the delta is large between the snapshots.
I will add the check as part of HDDS-7466.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] hemantk-12 commented on pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
hemantk-12 commented on PR #3885:
URL: https://github.com/apache/ozone/pull/3885#issuecomment-1314631472
LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] prashantpogde commented on pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
prashantpogde commented on PR #3885:
URL: https://github.com/apache/ozone/pull/3885#issuecomment-1317515472
LGTM for this set of changes. Please also file a Jira to track persistence and pagination logic when the diff set is too large to fit in memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] nandakumar131 commented on a diff in pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
nandakumar131 commented on code in PR #3885:
URL: https://github.com/apache/ozone/pull/3885#discussion_r1022481146
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java:
##########
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot;
+
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.snapshot.SnapshotDiffReport.DiffType;
+import org.apache.hadoop.ozone.om.snapshot.SnapshotDiffReport.DiffReportEntry;
+
+import org.apache.ozone.rocksdb.util.ManagedSstFileReader;
+import org.apache.ozone.rocksdb.util.RdbUtil;
+import org.rocksdb.RocksDBException;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * Class to generate snapshot diff.
+ */
+public class SnapshotDiffManager {
+
+ public SnapshotDiffReport getSnapshotDiffReport(final String volume,
+ final String bucket,
+ final OmSnapshot fromSnapshot,
+ final OmSnapshot toSnapshot)
+ throws IOException, RocksDBException {
+
+ // TODO: Once RocksDBCheckpointDiffer exposes method to get list
+ // of delta SST files, plug it in here.
+
+ Set<String> fromSnapshotFiles = RdbUtil.getKeyTableSSTFiles(fromSnapshot
+ .getMetadataManager().getStore().getDbLocation().getPath());
+ Set<String> toSnapshotFiles = RdbUtil.getKeyTableSSTFiles(toSnapshot
+ .getMetadataManager().getStore().getDbLocation().getPath());
+
+ final Set<String> deltaFiles = new HashSet<>();
+ deltaFiles.addAll(fromSnapshotFiles);
+ deltaFiles.addAll(toSnapshotFiles);
+
+ // TODO: Filter out the files.
+
+ final Stream<String> keysToCheck = new ManagedSstFileReader(deltaFiles)
+ .getKeyStream();
+
+ final BucketLayout bucketLayout = getBucketLayout(volume, bucket,
+ fromSnapshot.getMetadataManager());
+
+ final Table<String, OmKeyInfo> fsKeyTable = fromSnapshot
+ .getMetadataManager().getKeyTable(bucketLayout);
+ final Table<String, OmKeyInfo> tsKeyTable = toSnapshot
+ .getMetadataManager().getKeyTable(bucketLayout);
+
+ /*
+ * The reason for having ObjectID to KeyName mapping instead of OmKeyInfo
+ * is to reduce the memory footprint.
+ */
+ final Map<Long, String> oldObjIdToKeyMap = new HashMap<>();
+ // Long --> const. length
+ // String --> var. length "/dir1/dir2/dir3/dir4/dir5/key1"
+ final Map<Long, String> newObjIdToKeyMap = new HashMap<>();
+
Review Comment:
Thansk @GeorgeJahad for taking a look at the change.
The HashMaps only stores the keys which are affected between snapshots (created, deleted, renamed)
For create, we only add it to newObjIdToKeyMap
For delete, we only add it to oldObjIdToKeyMap
For rename, we add it to both newObjIdToKeyMap and oldObjIdToKeyMap
We don't store the whole key metadata here in these HashMaps, we only store the object ID and key name.
Object ID is a long value which is of size 64-bit.
We will reach 100 bytes for an entry only if all the key paths are of length ~99 characters.
So we will use 1 gig of memory per HashMap only if we have 10 million renames with an average key size of ~99 characters.
In this case, our SnapshotDiff report will also be around 2 gig.
A single RPC call cannot handle 2 gig of data, so SnapshotDiff for large delta will fail.
We don't have paging logic built into SnapshotDiff yet. We need support for partial SnapshotDiff generation and paging logic at the client side to support large SnapshotDiff.
For now, we can add a check while we generate the SnapshotDiff and throw an error if the delta is large between the snapshots.
I will add the check as part of HDDS-7466.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] nandakumar131 commented on a diff in pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
nandakumar131 commented on code in PR #3885:
URL: https://github.com/apache/ozone/pull/3885#discussion_r1016879347
##########
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdb/util/ManagedSstFileReader.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ozone.rocksdb.util;
+
+import org.rocksdb.Options;
+import org.rocksdb.ReadOptions;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.SstFileReader;
+import org.rocksdb.SstFileReaderIterator;
+
+import java.io.Closeable;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.NoSuchElementException;
+import java.util.Spliterator;
+import java.util.Spliterators;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+/**
+ * ManagedSSTFileReader provides an abstraction layer using which we can
+ * iterate over multiple underlying SST files transparently.
+ */
+public class ManagedSstFileReader {
+
+ private final Collection<String> sstFiles;
+
+ public ManagedSstFileReader(final Collection<String> sstFiles) {
+ this.sstFiles = sstFiles;
+ }
+ public Stream<String> getKeyStream() throws RocksDBException {
+ final OmSstFileIterator iterator = new OmSstFileIterator(sstFiles);
+ final Spliterator<String> spliterator = Spliterators
+ .spliteratorUnknownSize(iterator, 0);
+ return StreamSupport.stream(spliterator, false).onClose(iterator::close);
+ }
+
+ private static final class OmSstFileIterator implements
+ Iterator<String>, Closeable {
+
+ private final Iterator<String> fileNameIterator;
+ private final Options options;
+ private final ReadOptions readOptions;
+ private String currentFile;
+ private SstFileReader currentFileReader;
+ private SstFileReaderIterator currentFileIterator;
+
+ private OmSstFileIterator(Collection<String> files)
+ throws RocksDBException {
+ // TODO: Check if default Options and ReadOptions is enough.
+ this.options = new Options();
+ this.readOptions = new ReadOptions();
Review Comment:
Options is to control the behavior of the _database_.
ReadOptions is to control the behavior of the _get_ method call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] nandakumar131 commented on a diff in pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
nandakumar131 commented on code in PR #3885:
URL: https://github.com/apache/ozone/pull/3885#discussion_r1022462148
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -217,4 +224,40 @@ public static boolean isSnapshotKey(String[] keyParts) {
return (keyParts.length > 1) &&
(keyParts[0].compareTo(OM_SNAPSHOT_INDICATOR) == 0);
}
+
+ public SnapshotDiffReport getSnapshotDiffReport(final String volume,
+ final String bucket,
+ final String fromSnapshot,
+ final String toSnapshot)
+ throws IOException {
+ // Validate fromSnapshot and toSnapshot
+ final SnapshotInfo fsInfo = getSnapshotInfo(volume, bucket, fromSnapshot);
+ final SnapshotInfo tsInfo = getSnapshotInfo(volume, bucket, toSnapshot);
Review Comment:
Currently, we are not supporting diff against active filesystem.
The active filesystem will keep changing and we cannot take a lock on the filesystem while we iterate through the keys to calculate the diffs.
If we plan on supporting diff against the active filesystem, we need to write a design doc and have a discussion around it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] smengcl commented on pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
smengcl commented on PR #3885:
URL: https://github.com/apache/ozone/pull/3885#issuecomment-1302622174
Merged latest snapshot dev branch to this PR. Resolved a minor import conflict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] nandakumar131 commented on pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
nandakumar131 commented on PR #3885:
URL: https://github.com/apache/ozone/pull/3885#issuecomment-1297539240
This is still a work in progress. I will try to do some manual testing before making this PR ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GeorgeJahad commented on pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
GeorgeJahad commented on PR #3885:
URL: https://github.com/apache/ozone/pull/3885#issuecomment-1315655943
> We need support for partial SnapshotDiff generation
my question is really about partial SnapshotDiff generation. The current design requires all objectIds/keys to remain in memory concurrently, doesn't it? So how were we thinking of doing a partial diff?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] nandakumar131 commented on a diff in pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
nandakumar131 commented on code in PR #3885:
URL: https://github.com/apache/ozone/pull/3885#discussion_r1009758171
##########
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdb/util/ManagedSSTFileReader.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ozone.rocksdb.util;
+
+import org.rocksdb.Options;
+import org.rocksdb.ReadOptions;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.SstFileReader;
+import org.rocksdb.SstFileReaderIterator;
+
+import java.io.Closeable;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.NoSuchElementException;
+import java.util.Spliterator;
+import java.util.Spliterators;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+public class ManagedSSTFileReader {
+
+ private final Collection<String> sstFiles;
+
+ public ManagedSSTFileReader(final Collection<String> sstFiles) {
+ this.sstFiles = sstFiles;
+ }
+ public Stream<String> getKeyStream() throws RocksDBException {
+ final OMSSTFileIterator iterator = new OMSSTFileIterator(sstFiles);
+ final Spliterator<String> spliterator = Spliterators
+ .spliteratorUnknownSize(iterator, 0);
+ return StreamSupport.stream(spliterator, false).onClose(iterator::close);
+ }
+
+ public interface ClosableIterator<T> extends Iterator<T>, Closeable {}
Review Comment:
Directly implementing makes sense, not sure what I was thinking 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] nandakumar131 commented on a diff in pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
nandakumar131 commented on code in PR #3885:
URL: https://github.com/apache/ozone/pull/3885#discussion_r1009763752
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot;
+
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.snapshot.SnapshotDiffReport.DiffType;
+import org.apache.hadoop.ozone.om.snapshot.SnapshotDiffReport.DiffReportEntry;
+
+import org.apache.ozone.rocksdb.util.ManagedSSTFileReader;
+import org.rocksdb.RocksDBException;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+
+public class SnapshotDiffManager {
+
+ public SnapshotDiffReport getSnapshotDiffReport(final String volume,
+ final String bucket,
+ final OmSnapshot fromSnapshot,
+ final OmSnapshot toSnapshot)
+ throws IOException, RocksDBException {
+
+ // TODO: Once RocksDBCheckpointDiffer exposes method to get list
+ // of delta SST files, plug it in here.
+
+ final Set<String> deltaFiles = Collections.emptySet();
+
+ // TODO: Filter out the files.
+
+ final Stream<String> keysToCheck = new ManagedSSTFileReader(deltaFiles)
+ .getKeyStream();
+
+ final BucketLayout bucketLayout = getBucketLayout(volume, bucket,
+ fromSnapshot.getMetadataManager());
+
+ final Table<String, OmKeyInfo> fsKeyTable = fromSnapshot
+ .getMetadataManager().getKeyTable(bucketLayout);
+ final Table<String, OmKeyInfo> tsKeyTable = toSnapshot
+ .getMetadataManager().getKeyTable(bucketLayout);
+
+ /*
+ * The reason for having ObjectID to KeyName mapping instead of OmKeyInfo
+ * is to reduce the memory footprint.
+ */
+ final Map<Long, String> oldObjIdToKeyMap = new HashMap<>();
+ // Long --> const. length
+ // String --> var. length "/dir1/dir2/dir3/dir4/dir5/key1"
+ final Map<Long, String> newObjIdToKeyMap = new HashMap<>();
+
+ final Set<Long> objectIDsToCheck = new HashSet<>();
+
+ keysToCheck.forEach(key -> {
+ try {
+ final OmKeyInfo oldKey = fsKeyTable.get(key);
+ final OmKeyInfo newKey = tsKeyTable.get(key);
+ if (areKeysEqual(oldKey, newKey)) {
+ // We don't have to do anything.
+ return;
+ }
+ if (oldKey != null) {
+ final long oldObjId = oldKey.getObjectID();
+ oldObjIdToKeyMap.put(oldObjId, oldKey.getKeyName());
+ objectIDsToCheck.add(oldObjId);
+ }
+ if (newKey != null) {
+ final long newObjId = newKey.getObjectID();
+ newObjIdToKeyMap.put(newObjId, newKey.getKeyName());
+ objectIDsToCheck.add(newObjId);
+ }
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ keysToCheck.close();
+
+ return new SnapshotDiffReport(volume, bucket, fromSnapshot.getName(),
+ toSnapshot.getName(), generateDiffReport(objectIDsToCheck,
+ oldObjIdToKeyMap, newObjIdToKeyMap));
+ }
+
+ private List<DiffReportEntry> generateDiffReport(
+ final Set<Long> objectIDsToCheck,
+ final Map<Long, String> oldObjIdToKeyMap,
+ final Map<Long, String> newObjIdToKeyMap) {
+
+ final List<DiffReportEntry> deleteDiffs = new ArrayList<>();
+ final List<DiffReportEntry> renameDiffs = new ArrayList<>();
+ final List<DiffReportEntry> createDiffs = new ArrayList<>();
+ final List<DiffReportEntry> modifyDiffs = new ArrayList<>();
+
+
+ for (Long id : objectIDsToCheck) {
+ /*
+ * This key can be
+ * -> Created after the old snapshot was taken, which means it will be
+ * missing in oldKeyTable and present in newKeyTable.
+ * -> Deleted after the old snapshot was taken, which means it will be
+ * present in oldKeyTable and missing in newKeyTable.
+ * -> Modified after the old snapshot was taken, which means it will be
+ * present in oldKeyTable and present in newKeyTable with same
+ * Object ID but with different metadata.
+ * -> Renamed after the old snapshot was taken, which means it will be
+ * present in oldKeyTable and present in newKeyTable but with different
+ * name and same Object ID.
+ */
+
+ final String oldKeyName = oldObjIdToKeyMap.get(id);
+ final String newKeyName = newObjIdToKeyMap.get(id);
+
+ if (oldKeyName == null && newKeyName == null) {
+ // This cannot happen.
+ continue;
+ }
+
+ // Key Created.
+ if (oldKeyName == null) {
+ createDiffs.add(DiffReportEntry.of(DiffType.CREATE, newKeyName));
+ continue;
+ }
+
+ // Key Deleted.
+ if(newKeyName == null) {
+ deleteDiffs.add(DiffReportEntry.of(DiffType.DELETE, oldKeyName));
+ continue;
+ }
+
+ // Key modified.
+ if(oldKeyName.equals(newKeyName)) {
+ modifyDiffs.add(DiffReportEntry.of(DiffType.MODIFY, newKeyName));
+ continue;
+ }
+
+ // Key Renamed.
+ renameDiffs.add(DiffReportEntry.of(DiffType.RENAME,
+ oldKeyName, newKeyName));
+ }
+ /*
+ * The order in which snap-diff should be applied
+ *
+ * 1. Delete diffs
+ * 2. Rename diffs
+ * 3. Create diffs
+ * 4. Modified diffs
+ *
+ * Consider the following scenario
+ *
+ * 1. File "A" is created.
+ * 2. File "B" is created.
+ * 3. File "C" is created.
+ * Snapshot "1" is taken.
+ *
+ * Case 1:
+ * 1. File "A" is deleted.
+ * 2. File "B" is renamed to "A".
+ * Snapshot "2" is taken.
+ *
+ * Snapshot diff should be applied in the following order:
+ * 1. Delete "A"
+ * 2. Rename "B" to "A"
+ *
+ *
+ * Case 2:
+ * 1. File "B" is renamed to "C".
+ * 2. File "B" is created.
+ * Snapshot "2" is taken.
+ *
+ * Snapshot diff should be applied in the following order:
+ * 1. Rename "B" to "C"
+ * 2. Create "B"
+ *
+ */
+
+ final List<DiffReportEntry> snapshotDiffs = new ArrayList<>();
+ snapshotDiffs.addAll(deleteDiffs);
+ snapshotDiffs.addAll(renameDiffs);
+ snapshotDiffs.addAll(createDiffs);
+ snapshotDiffs.addAll(modifyDiffs);
+ return snapshotDiffs;
+ }
+
+ private BucketLayout getBucketLayout(final String volume,
+ final String bucket,
+ final OMMetadataManager mManager)
+ throws IOException {
+ final String bucketTableKey = mManager.getBucketKey(volume, bucket);
+ return mManager.getBucketTable().get(bucketTableKey).getBucketLayout();
Review Comment:
`mManager.getBucketKey(volume, bucket)` can never return null.
`mManager.getBucketTable().get(bucketTableKey)` should not return null once we implement proper locking logic in the snapshot code. For now, it is theoretically possible to get a null pointer exception here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] nandakumar131 commented on pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
nandakumar131 commented on PR #3885:
URL: https://github.com/apache/ozone/pull/3885#issuecomment-1316607692
> So how were we thinking of doing a partial diff?
If we plan on implementing partial diff, then we have to make `newObjIdToKeyMap` and `oldObjIdToKeyMap` as some kind of disk-based Map (maybe we can use RocksDB?). We should also start writing the SnapshotDiff output to some file on disk.
We can explore more on this when we work on implementing the paging logic ( & support huge SnapshotDiff).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] nandakumar131 commented on a diff in pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
nandakumar131 commented on code in PR #3885:
URL: https://github.com/apache/ozone/pull/3885#discussion_r1009761163
##########
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdb/util/ManagedSSTFileReader.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ozone.rocksdb.util;
+
+import org.rocksdb.Options;
+import org.rocksdb.ReadOptions;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.SstFileReader;
+import org.rocksdb.SstFileReaderIterator;
+
+import java.io.Closeable;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.NoSuchElementException;
+import java.util.Spliterator;
+import java.util.Spliterators;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+public class ManagedSSTFileReader {
+
+ private final Collection<String> sstFiles;
+
+ public ManagedSSTFileReader(final Collection<String> sstFiles) {
+ this.sstFiles = sstFiles;
+ }
+ public Stream<String> getKeyStream() throws RocksDBException {
+ final OMSSTFileIterator iterator = new OMSSTFileIterator(sstFiles);
+ final Spliterator<String> spliterator = Spliterators
+ .spliteratorUnknownSize(iterator, 0);
+ return StreamSupport.stream(spliterator, false).onClose(iterator::close);
+ }
+
+ public interface ClosableIterator<T> extends Iterator<T>, Closeable {}
+
+ private static class OMSSTFileIterator implements ClosableIterator<String> {
+
+ private final Iterator<String> fileNameIterator;
+ private final Options options;
+ private final ReadOptions readOptions;
+ private String currentFile;
+ private SstFileReader currentFileReader;
+ private SstFileReaderIterator currentFileIterator;
+
+ private OMSSTFileIterator(Collection<String> files) throws RocksDBException {
+ // TODO: Check if default Options and ReadOptions is enough.
+ this.options = new Options();
+ this.readOptions = new ReadOptions();
+ this.fileNameIterator = files.iterator();
+ moveToNextFile();
+ }
+
+ @Override
+ public boolean hasNext() {
+ try {
+ do {
+ if (currentFileIterator.isValid()) {
+ return true;
+ }
+ } while (moveToNextFile());
+ } catch (RocksDBException e) {
+ return false;
Review Comment:
Added RuntimeException here for now.
We have to catch and handle this exception in the caller, we should not crash OM process because of this exception.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -196,4 +203,40 @@ public static boolean isSnapshotKey(String[] keyParts) {
return (keyParts.length > 1) &&
(keyParts[0].compareTo(OM_SNAPSHOT_INDICATOR) == 0);
}
-}
+
+ public SnapshotDiffReport getSnapshotDiffReport(final String volume,
+ final String bucket,
+ final String fromSnapshot,
+ final String toSnapshot)
+ throws IOException {
+ // Validate fromSnapshot and toSnapshot
+ final SnapshotInfo fsInfo = getSnapshotInfo(volume, bucket, fromSnapshot);
+ final SnapshotInfo tsInfo = getSnapshotInfo(volume, bucket, toSnapshot);
+ verifySnapshotInfoForSnapDiff(fsInfo, tsInfo);
+
+ final String fsKey = SnapshotInfo.getTableKey(volume, bucket, fromSnapshot);
+ final String tsKey = SnapshotInfo.getTableKey(volume, bucket, toSnapshot);
+ try {
+ final OmSnapshot fs = snapshotCache.get(fsKey);
+ final OmSnapshot ts = snapshotCache.get(tsKey);
+ return snapshotDiffManager.getSnapshotDiffReport(volume, bucket, fs, ts);
+ } catch (ExecutionException | RocksDBException e) {
+ throw new IOException(e.getCause());
+ }
+ }
+
+ private void verifySnapshotInfoForSnapDiff(final SnapshotInfo fromSnapshot,
+ final SnapshotInfo toSnapshot)
+ throws IOException {
+ if ((fromSnapshot.getSnapshotStatus() != SnapshotStatus.SNAPSHOT_ACTIVE) ||
+ (toSnapshot.getSnapshotStatus() != SnapshotStatus.SNAPSHOT_ACTIVE)) {
+ // TODO: throw custom snapshot exception.
+ throw new IOException("Cannot generate snapshot diff for non-active " +
+ "snapshots.");
+ }
+ if (fromSnapshot.getCreationTime() > toSnapshot.getCreationTime()) {
+ throw new IOException("fromSnapshot should be older than to toSnapshot");
+ }
+ }
+
+ }
Review Comment:
Fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] nandakumar131 commented on a diff in pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
nandakumar131 commented on code in PR #3885:
URL: https://github.com/apache/ozone/pull/3885#discussion_r1009761911
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffReport.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot;
+
+import java.util.Collections;
+import java.util.List;
+
+public class SnapshotDiffReport {
+
+ private final static String LINE_SEPARATOR = System.getProperty(
+ "line.separator", "\n");
+
+ /**
+ * Types of the difference, which include CREATE, MODIFY, DELETE, and RENAME.
+ * Each type has a label for representation:
+ * + CREATE
+ * M MODIFY
+ * - DELETE
+ * R RENAME
+ */
+ public enum DiffType {
+ CREATE("+"),
+ MODIFY("M"),
+ DELETE("-"),
+ RENAME("R");
+
+ private final String label;
+
+ DiffType(String label) {
+ this.label = label;
+ }
+
+ public String getLabel() {
+ return label;
+ }
+ }
+
+ public static class DiffReportEntry {
+
+ /**
+ * The type of diff.
+ */
+ private final DiffType type;
+
+ /**
+ * Source File/Object path.
+ */
+ private final String sourcePath;
+
+ /**
+ * Destination File/Object path, if this is a re-name operation.
+ */
+ private final String targetPath;
+
+ private DiffReportEntry(final DiffType type, final String sourcePath,
+ final String targetPath) {
+ this.type = type;
+ this.sourcePath = sourcePath;
+ this.targetPath = targetPath;
+ }
+
+ public static DiffReportEntry of(final DiffType type,
+ final String sourcePath) {
+ return of(type, sourcePath, null);
+ }
+
+ public static DiffReportEntry of(final DiffType type,
+ final String sourcePath,
+ final String targetPath) {
+ return new DiffReportEntry(type, sourcePath, null);
Review Comment:
Fixed it. Thanks for catching this.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffReport.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot;
+
+import java.util.Collections;
+import java.util.List;
+
+public class SnapshotDiffReport {
+
+ private final static String LINE_SEPARATOR = System.getProperty(
+ "line.separator", "\n");
+
+ /**
+ * Types of the difference, which include CREATE, MODIFY, DELETE, and RENAME.
+ * Each type has a label for representation:
+ * + CREATE
+ * M MODIFY
+ * - DELETE
+ * R RENAME
+ */
+ public enum DiffType {
+ CREATE("+"),
+ MODIFY("M"),
+ DELETE("-"),
+ RENAME("R");
+
+ private final String label;
+
+ DiffType(String label) {
+ this.label = label;
+ }
+
+ public String getLabel() {
+ return label;
+ }
+ }
+
+ public static class DiffReportEntry {
+
+ /**
+ * The type of diff.
+ */
+ private final DiffType type;
+
+ /**
+ * Source File/Object path.
+ */
+ private final String sourcePath;
+
+ /**
+ * Destination File/Object path, if this is a re-name operation.
+ */
+ private final String targetPath;
+
+ private DiffReportEntry(final DiffType type, final String sourcePath,
+ final String targetPath) {
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] prashantpogde commented on a diff in pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
prashantpogde commented on code in PR #3885:
URL: https://github.com/apache/ozone/pull/3885#discussion_r1024386577
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -217,4 +224,40 @@ public static boolean isSnapshotKey(String[] keyParts) {
return (keyParts.length > 1) &&
(keyParts[0].compareTo(OM_SNAPSHOT_INDICATOR) == 0);
}
+
+ public SnapshotDiffReport getSnapshotDiffReport(final String volume,
+ final String bucket,
+ final String fromSnapshot,
+ final String toSnapshot)
+ throws IOException {
+ // Validate fromSnapshot and toSnapshot
+ final SnapshotInfo fsInfo = getSnapshotInfo(volume, bucket, fromSnapshot);
+ final SnapshotInfo tsInfo = getSnapshotInfo(volume, bucket, toSnapshot);
Review Comment:
There is no concept of snapdiff against active file system. We can do diff between only two stable(non-changing) images.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] nandakumar131 commented on a diff in pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
nandakumar131 commented on code in PR #3885:
URL: https://github.com/apache/ozone/pull/3885#discussion_r1016876329
##########
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdb/util/RdbUtil.java:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ozone.rocksdb.util;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksDBException;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Temporary class to test snapshot diff functionality.
+ * This should be removed later.
+ */
+public final class RdbUtil {
+
+ private RdbUtil() { }
+
+ public static Set<String> getKeyTableSSTFiles(final String dbLocation)
+ throws RocksDBException {
+ final List<ColumnFamilyHandle> columnFamilyHandles = new ArrayList<>();
+ final List<ColumnFamilyDescriptor> cfd = new ArrayList<>();
+ cfd.add(
+ new ColumnFamilyDescriptor(
+ "keyTable".getBytes(StandardCharsets.UTF_8)));
+ cfd.add(
+ new ColumnFamilyDescriptor(
+ "default".getBytes(StandardCharsets.UTF_8)));
+ try (DBOptions options = new DBOptions()) {
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] nandakumar131 commented on a diff in pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
nandakumar131 commented on code in PR #3885:
URL: https://github.com/apache/ozone/pull/3885#discussion_r1009757549
##########
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdb/util/ManagedSSTFileReader.java:
##########
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ozone.rocksdb.util;
+
+import org.rocksdb.Options;
+import org.rocksdb.ReadOptions;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.SstFileReader;
+import org.rocksdb.SstFileReaderIterator;
+
+import java.io.Closeable;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.NoSuchElementException;
+import java.util.Spliterator;
+import java.util.Spliterators;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+public class ManagedSSTFileReader {
Review Comment:
Fixed it. Thanks for the suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] hemantk-12 commented on a diff in pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
hemantk-12 commented on code in PR #3885:
URL: https://github.com/apache/ozone/pull/3885#discussion_r1010966508
##########
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdb/util/ManagedSstFileReader.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ozone.rocksdb.util;
+
+import org.rocksdb.Options;
+import org.rocksdb.ReadOptions;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.SstFileReader;
+import org.rocksdb.SstFileReaderIterator;
+
+import java.io.Closeable;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.NoSuchElementException;
+import java.util.Spliterator;
+import java.util.Spliterators;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+/**
+ * ManagedSSTFileReader provides an abstraction layer using which we can
Review Comment:
nit: `ManagedSstFileReader`
##########
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdb/util/RdbUtil.java:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ozone.rocksdb.util;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksDBException;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Temporary class to test snapshot diff functionality.
+ * This should be removed later.
+ */
+public final class RdbUtil {
+
+ private RdbUtil() { }
+
+ public static Set<String> getKeyTableSSTFiles(final String dbLocation)
+ throws RocksDBException {
+ final List<ColumnFamilyHandle> columnFamilyHandles = new ArrayList<>();
+ final List<ColumnFamilyDescriptor> cfd = new ArrayList<>();
+ cfd.add(
+ new ColumnFamilyDescriptor(
+ "keyTable".getBytes(StandardCharsets.UTF_8)));
+ cfd.add(
+ new ColumnFamilyDescriptor(
+ "default".getBytes(StandardCharsets.UTF_8)));
+ try (DBOptions options = new DBOptions()) {
Review Comment:
It doesn't need to be nested try. You can initialized multiple objects in single try like:
```
try (DBOptions options = new DBOptions();
RocksDB rocksDB = RocksDB.openReadOnly(options, dbLocation, cfd, columnFamilyHandles)) {
return rocksDB.getLiveFilesMetaData().stream()
.map(lfm -> new File(lfm.path(), lfm.fileName()).getPath())
.collect(Collectors.toCollection(HashSet::new));
}
```
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot;
+
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.snapshot.SnapshotDiffReport.DiffType;
+import org.apache.hadoop.ozone.om.snapshot.SnapshotDiffReport.DiffReportEntry;
+
+import org.apache.ozone.rocksdb.util.ManagedSSTFileReader;
+import org.rocksdb.RocksDBException;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+
+public class SnapshotDiffManager {
+
+ public SnapshotDiffReport getSnapshotDiffReport(final String volume,
+ final String bucket,
+ final OmSnapshot fromSnapshot,
+ final OmSnapshot toSnapshot)
+ throws IOException, RocksDBException {
+
+ // TODO: Once RocksDBCheckpointDiffer exposes method to get list
+ // of delta SST files, plug it in here.
+
+ final Set<String> deltaFiles = Collections.emptySet();
+
+ // TODO: Filter out the files.
+
+ final Stream<String> keysToCheck = new ManagedSSTFileReader(deltaFiles)
+ .getKeyStream();
+
+ final BucketLayout bucketLayout = getBucketLayout(volume, bucket,
+ fromSnapshot.getMetadataManager());
+
+ final Table<String, OmKeyInfo> fsKeyTable = fromSnapshot
+ .getMetadataManager().getKeyTable(bucketLayout);
+ final Table<String, OmKeyInfo> tsKeyTable = toSnapshot
+ .getMetadataManager().getKeyTable(bucketLayout);
+
+ /*
+ * The reason for having ObjectID to KeyName mapping instead of OmKeyInfo
+ * is to reduce the memory footprint.
+ */
+ final Map<Long, String> oldObjIdToKeyMap = new HashMap<>();
+ // Long --> const. length
+ // String --> var. length "/dir1/dir2/dir3/dir4/dir5/key1"
+ final Map<Long, String> newObjIdToKeyMap = new HashMap<>();
+
+ final Set<Long> objectIDsToCheck = new HashSet<>();
+
+ keysToCheck.forEach(key -> {
+ try {
+ final OmKeyInfo oldKey = fsKeyTable.get(key);
+ final OmKeyInfo newKey = tsKeyTable.get(key);
+ if (areKeysEqual(oldKey, newKey)) {
+ // We don't have to do anything.
+ return;
+ }
+ if (oldKey != null) {
+ final long oldObjId = oldKey.getObjectID();
+ oldObjIdToKeyMap.put(oldObjId, oldKey.getKeyName());
+ objectIDsToCheck.add(oldObjId);
+ }
+ if (newKey != null) {
+ final long newObjId = newKey.getObjectID();
+ newObjIdToKeyMap.put(newObjId, newKey.getKeyName());
+ objectIDsToCheck.add(newObjId);
+ }
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ });
+ keysToCheck.close();
+
+ return new SnapshotDiffReport(volume, bucket, fromSnapshot.getName(),
+ toSnapshot.getName(), generateDiffReport(objectIDsToCheck,
+ oldObjIdToKeyMap, newObjIdToKeyMap));
+ }
+
+ private List<DiffReportEntry> generateDiffReport(
+ final Set<Long> objectIDsToCheck,
+ final Map<Long, String> oldObjIdToKeyMap,
+ final Map<Long, String> newObjIdToKeyMap) {
+
+ final List<DiffReportEntry> deleteDiffs = new ArrayList<>();
+ final List<DiffReportEntry> renameDiffs = new ArrayList<>();
+ final List<DiffReportEntry> createDiffs = new ArrayList<>();
+ final List<DiffReportEntry> modifyDiffs = new ArrayList<>();
+
+
+ for (Long id : objectIDsToCheck) {
+ /*
+ * This key can be
+ * -> Created after the old snapshot was taken, which means it will be
+ * missing in oldKeyTable and present in newKeyTable.
+ * -> Deleted after the old snapshot was taken, which means it will be
+ * present in oldKeyTable and missing in newKeyTable.
+ * -> Modified after the old snapshot was taken, which means it will be
+ * present in oldKeyTable and present in newKeyTable with same
+ * Object ID but with different metadata.
+ * -> Renamed after the old snapshot was taken, which means it will be
+ * present in oldKeyTable and present in newKeyTable but with different
+ * name and same Object ID.
+ */
+
+ final String oldKeyName = oldObjIdToKeyMap.get(id);
+ final String newKeyName = newObjIdToKeyMap.get(id);
+
+ if (oldKeyName == null && newKeyName == null) {
+ // This cannot happen.
+ continue;
+ }
+
+ // Key Created.
+ if (oldKeyName == null) {
+ createDiffs.add(DiffReportEntry.of(DiffType.CREATE, newKeyName));
+ continue;
+ }
+
+ // Key Deleted.
+ if(newKeyName == null) {
+ deleteDiffs.add(DiffReportEntry.of(DiffType.DELETE, oldKeyName));
+ continue;
+ }
+
+ // Key modified.
+ if(oldKeyName.equals(newKeyName)) {
+ modifyDiffs.add(DiffReportEntry.of(DiffType.MODIFY, newKeyName));
+ continue;
+ }
+
+ // Key Renamed.
+ renameDiffs.add(DiffReportEntry.of(DiffType.RENAME,
+ oldKeyName, newKeyName));
+ }
+ /*
+ * The order in which snap-diff should be applied
+ *
+ * 1. Delete diffs
+ * 2. Rename diffs
+ * 3. Create diffs
+ * 4. Modified diffs
+ *
+ * Consider the following scenario
+ *
+ * 1. File "A" is created.
+ * 2. File "B" is created.
+ * 3. File "C" is created.
+ * Snapshot "1" is taken.
+ *
+ * Case 1:
+ * 1. File "A" is deleted.
+ * 2. File "B" is renamed to "A".
+ * Snapshot "2" is taken.
+ *
+ * Snapshot diff should be applied in the following order:
+ * 1. Delete "A"
+ * 2. Rename "B" to "A"
+ *
+ *
+ * Case 2:
+ * 1. File "B" is renamed to "C".
+ * 2. File "B" is created.
+ * Snapshot "2" is taken.
+ *
+ * Snapshot diff should be applied in the following order:
+ * 1. Rename "B" to "C"
+ * 2. Create "B"
+ *
+ */
+
+ final List<DiffReportEntry> snapshotDiffs = new ArrayList<>();
+ snapshotDiffs.addAll(deleteDiffs);
+ snapshotDiffs.addAll(renameDiffs);
+ snapshotDiffs.addAll(createDiffs);
+ snapshotDiffs.addAll(modifyDiffs);
+ return snapshotDiffs;
+ }
+
+ private BucketLayout getBucketLayout(final String volume,
+ final String bucket,
+ final OMMetadataManager mManager)
+ throws IOException {
+ final String bucketTableKey = mManager.getBucketKey(volume, bucket);
+ return mManager.getBucketTable().get(bucketTableKey).getBucketLayout();
Review Comment:
Thanks for clarification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
GeorgeJahad commented on code in PR #3885:
URL: https://github.com/apache/ozone/pull/3885#discussion_r1018514954
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java:
##########
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot;
+
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.snapshot.SnapshotDiffReport.DiffType;
+import org.apache.hadoop.ozone.om.snapshot.SnapshotDiffReport.DiffReportEntry;
+
+import org.apache.ozone.rocksdb.util.ManagedSstFileReader;
+import org.apache.ozone.rocksdb.util.RdbUtil;
+import org.rocksdb.RocksDBException;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * Class to generate snapshot diff.
+ */
+public class SnapshotDiffManager {
+
+ public SnapshotDiffReport getSnapshotDiffReport(final String volume,
+ final String bucket,
+ final OmSnapshot fromSnapshot,
+ final OmSnapshot toSnapshot)
+ throws IOException, RocksDBException {
+
+ // TODO: Once RocksDBCheckpointDiffer exposes method to get list
+ // of delta SST files, plug it in here.
+
+ Set<String> fromSnapshotFiles = RdbUtil.getKeyTableSSTFiles(fromSnapshot
+ .getMetadataManager().getStore().getDbLocation().getPath());
+ Set<String> toSnapshotFiles = RdbUtil.getKeyTableSSTFiles(toSnapshot
+ .getMetadataManager().getStore().getDbLocation().getPath());
+
+ final Set<String> deltaFiles = new HashSet<>();
+ deltaFiles.addAll(fromSnapshotFiles);
+ deltaFiles.addAll(toSnapshotFiles);
+
+ // TODO: Filter out the files.
+
+ final Stream<String> keysToCheck = new ManagedSstFileReader(deltaFiles)
+ .getKeyStream();
+
+ final BucketLayout bucketLayout = getBucketLayout(volume, bucket,
+ fromSnapshot.getMetadataManager());
+
+ final Table<String, OmKeyInfo> fsKeyTable = fromSnapshot
+ .getMetadataManager().getKeyTable(bucketLayout);
+ final Table<String, OmKeyInfo> tsKeyTable = toSnapshot
+ .getMetadataManager().getKeyTable(bucketLayout);
+
+ /*
+ * The reason for having ObjectID to KeyName mapping instead of OmKeyInfo
+ * is to reduce the memory footprint.
+ */
+ final Map<Long, String> oldObjIdToKeyMap = new HashMap<>();
+ // Long --> const. length
+ // String --> var. length "/dir1/dir2/dir3/dir4/dir5/key1"
+ final Map<Long, String> newObjIdToKeyMap = new HashMap<>();
+
Review Comment:
It looks like these hashmaps need to simultaneously store every key being diffed. Is that correct?
So if each entry is 100 bytes, and there are 10 million keys being diffed, that is 1 gig of memory per hashmap. Is that acceptable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] nandakumar131 merged pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
nandakumar131 merged PR #3885:
URL: https://github.com/apache/ozone/pull/3885
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] prashantpogde commented on a diff in pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
prashantpogde commented on code in PR #3885:
URL: https://github.com/apache/ozone/pull/3885#discussion_r1024389507
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java:
##########
@@ -0,0 +1,235 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.om.snapshot;
+
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.ozone.om.OMMetadataManager;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.helpers.BucketLayout;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.snapshot.SnapshotDiffReport.DiffType;
+import org.apache.hadoop.ozone.om.snapshot.SnapshotDiffReport.DiffReportEntry;
+
+import org.apache.ozone.rocksdb.util.ManagedSstFileReader;
+import org.apache.ozone.rocksdb.util.RdbUtil;
+import org.rocksdb.RocksDBException;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * Class to generate snapshot diff.
+ */
+public class SnapshotDiffManager {
+
+ public SnapshotDiffReport getSnapshotDiffReport(final String volume,
+ final String bucket,
+ final OmSnapshot fromSnapshot,
+ final OmSnapshot toSnapshot)
+ throws IOException, RocksDBException {
+
+ // TODO: Once RocksDBCheckpointDiffer exposes method to get list
+ // of delta SST files, plug it in here.
+
+ Set<String> fromSnapshotFiles = RdbUtil.getKeyTableSSTFiles(fromSnapshot
+ .getMetadataManager().getStore().getDbLocation().getPath());
+ Set<String> toSnapshotFiles = RdbUtil.getKeyTableSSTFiles(toSnapshot
+ .getMetadataManager().getStore().getDbLocation().getPath());
+
+ final Set<String> deltaFiles = new HashSet<>();
+ deltaFiles.addAll(fromSnapshotFiles);
+ deltaFiles.addAll(toSnapshotFiles);
+
+ // TODO: Filter out the files.
+
+ final Stream<String> keysToCheck = new ManagedSstFileReader(deltaFiles)
+ .getKeyStream();
+
+ final BucketLayout bucketLayout = getBucketLayout(volume, bucket,
+ fromSnapshot.getMetadataManager());
+
+ final Table<String, OmKeyInfo> fsKeyTable = fromSnapshot
+ .getMetadataManager().getKeyTable(bucketLayout);
+ final Table<String, OmKeyInfo> tsKeyTable = toSnapshot
+ .getMetadataManager().getKeyTable(bucketLayout);
+
+ /*
+ * The reason for having ObjectID to KeyName mapping instead of OmKeyInfo
+ * is to reduce the memory footprint.
+ */
+ final Map<Long, String> oldObjIdToKeyMap = new HashMap<>();
+ // Long --> const. length
+ // String --> var. length "/dir1/dir2/dir3/dir4/dir5/key1"
+ final Map<Long, String> newObjIdToKeyMap = new HashMap<>();
+
Review Comment:
Even with the pagination logic, the map has to be available for the entire diff set in memory. Let us file another Jira to track this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] nandakumar131 commented on a diff in pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
nandakumar131 commented on code in PR #3885:
URL: https://github.com/apache/ozone/pull/3885#discussion_r1009761422
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -196,4 +203,40 @@ public static boolean isSnapshotKey(String[] keyParts) {
return (keyParts.length > 1) &&
(keyParts[0].compareTo(OM_SNAPSHOT_INDICATOR) == 0);
}
-}
+
+ public SnapshotDiffReport getSnapshotDiffReport(final String volume,
+ final String bucket,
+ final String fromSnapshot,
+ final String toSnapshot)
+ throws IOException {
+ // Validate fromSnapshot and toSnapshot
+ final SnapshotInfo fsInfo = getSnapshotInfo(volume, bucket, fromSnapshot);
+ final SnapshotInfo tsInfo = getSnapshotInfo(volume, bucket, toSnapshot);
+ verifySnapshotInfoForSnapDiff(fsInfo, tsInfo);
+
+ final String fsKey = SnapshotInfo.getTableKey(volume, bucket, fromSnapshot);
+ final String tsKey = SnapshotInfo.getTableKey(volume, bucket, toSnapshot);
+ try {
+ final OmSnapshot fs = snapshotCache.get(fsKey);
+ final OmSnapshot ts = snapshotCache.get(tsKey);
+ return snapshotDiffManager.getSnapshotDiffReport(volume, bucket, fs, ts);
+ } catch (ExecutionException | RocksDBException e) {
+ throw new IOException(e.getCause());
+ }
+ }
+
+ private void verifySnapshotInfoForSnapDiff(final SnapshotInfo fromSnapshot,
+ final SnapshotInfo toSnapshot)
+ throws IOException {
+ if ((fromSnapshot.getSnapshotStatus() != SnapshotStatus.SNAPSHOT_ACTIVE) ||
+ (toSnapshot.getSnapshotStatus() != SnapshotStatus.SNAPSHOT_ACTIVE)) {
+ // TODO: throw custom snapshot exception.
+ throw new IOException("Cannot generate snapshot diff for non-active " +
+ "snapshots.");
+ }
+ if (fromSnapshot.getCreationTime() > toSnapshot.getCreationTime()) {
+ throw new IOException("fromSnapshot should be older than to toSnapshot");
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] DaveTeng0 commented on a diff in pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
DaveTeng0 commented on code in PR #3885:
URL: https://github.com/apache/ozone/pull/3885#discussion_r1010710274
##########
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdb/util/ManagedSstFileReader.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ozone.rocksdb.util;
+
+import org.rocksdb.Options;
+import org.rocksdb.ReadOptions;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.SstFileReader;
+import org.rocksdb.SstFileReaderIterator;
+
+import java.io.Closeable;
+import java.util.Collection;
+import java.util.Iterator;
+import java.util.NoSuchElementException;
+import java.util.Spliterator;
+import java.util.Spliterators;
+import java.util.stream.Stream;
+import java.util.stream.StreamSupport;
+
+/**
+ * ManagedSSTFileReader provides an abstraction layer using which we can
+ * iterate over multiple underlying SST files transparently.
+ */
+public class ManagedSstFileReader {
+
+ private final Collection<String> sstFiles;
+
+ public ManagedSstFileReader(final Collection<String> sstFiles) {
+ this.sstFiles = sstFiles;
+ }
+ public Stream<String> getKeyStream() throws RocksDBException {
+ final OmSstFileIterator iterator = new OmSstFileIterator(sstFiles);
+ final Spliterator<String> spliterator = Spliterators
+ .spliteratorUnknownSize(iterator, 0);
+ return StreamSupport.stream(spliterator, false).onClose(iterator::close);
+ }
+
+ private static final class OmSstFileIterator implements
+ Iterator<String>, Closeable {
+
+ private final Iterator<String> fileNameIterator;
+ private final Options options;
+ private final ReadOptions readOptions;
+ private String currentFile;
+ private SstFileReader currentFileReader;
+ private SstFileReaderIterator currentFileIterator;
+
+ private OmSstFileIterator(Collection<String> files)
+ throws RocksDBException {
+ // TODO: Check if default Options and ReadOptions is enough.
+ this.options = new Options();
+ this.readOptions = new ReadOptions();
Review Comment:
just for myself learning~ what's difference between Options & ReadOptions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] nandakumar131 commented on pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
nandakumar131 commented on PR #3885:
URL: https://github.com/apache/ozone/pull/3885#issuecomment-1307528228
This PR doesn't have additional unit tests to cover snapshot diff logic. Created [HDDS-7467](https://issues.apache.org/jira/browse/HDDS-7467) to add unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #3885: HDDS-7410. Ozone snapshot diff skeleton code.
Posted by GitBox <gi...@apache.org>.
GeorgeJahad commented on code in PR #3885:
URL: https://github.com/apache/ozone/pull/3885#discussion_r1018490593
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -217,4 +224,40 @@ public static boolean isSnapshotKey(String[] keyParts) {
return (keyParts.length > 1) &&
(keyParts[0].compareTo(OM_SNAPSHOT_INDICATOR) == 0);
}
+
+ public SnapshotDiffReport getSnapshotDiffReport(final String volume,
+ final String bucket,
+ final String fromSnapshot,
+ final String toSnapshot)
+ throws IOException {
+ // Validate fromSnapshot and toSnapshot
+ final SnapshotInfo fsInfo = getSnapshotInfo(volume, bucket, fromSnapshot);
+ final SnapshotInfo tsInfo = getSnapshotInfo(volume, bucket, toSnapshot);
Review Comment:
This seems to indicate that the diff can not be against the active filesystem. Is that going to be supported?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org