You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ozone.apache.org by si...@apache.org on 2023/02/02 05:19:12 UTC
[ozone] branch master updated (1a796f1b54 -> f01cebe19b)
This is an automated email from the ASF dual-hosted git repository.
siyao pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/ozone.git
from 1a796f1b54 HDDS-7821. Let push replication use compression from configuration (#4229)
add 2410551d2e HDDS-6849. OM changes for tracking Snapshot Metadata. (#3582)
add a0a9d221d0 Merge branch 'master' into HDDS-6517-Snapshot
add 4dba9020ed HDDS-6853. [Snapshot] Implement Create Snapshot API. (#3652)
add eafd2ccec0 HDDS-6964. [Snapshot] Split out shared "Path based access" code from OM. (#3653)
add 03dfa243ac HDDS-7224. Create a new RocksDBCheckpoint Diff utility. (#3755)
add 6eb19a7710 HDDS-6855. [SNAPSHOTS] Path Based Access for Ozone Snapshots. (#3729)
add 4be3864675 Merge branch 'master' into HDDS-6517-Snapshot
add f30c0621ea HDDS-6983. Snapshot Chain - list of snapshots per snapshottable bucket (#3658)
add 9538954e4b HDDS-7427. [Snapshot] Add unit-testcases for Ozone snapshot feature. (#3896)
add e642ddeb55 HDDS-7281. [Snapshot] Handle RocksDB compaction DAG persistence and reconstruction (#3824)
add d1a834ce09 HDDS-7452. [snapshot] Add unit-testcases for snapshot create validation. (#3946)
add f1f1c0ba42 HDDS-7375. [Snapshot] Implement `ozone fs` command for creating snapshot (#3937)
add 83072e0f8d HDDS-7410. Ozone snapshot diff skeleton code. (#3885)
add 06ae2fe8d5 HDDS-7494. [Snapshot] Fix SnapshotInfo#dbTxSequenceNumber (de)serialization (#3965)
add 67e67250bd HDDS-6854. [Snapshot] Implement List Snapshot API (#3784)
add 6fe2ab8011 HDDS-7279. Snapshot Create requires Double Buffer Flush thread to split the commit batch. (#3958)
add 1a52ae199a HDDS-7571. [snapshot] Add unit-testcases for Ozone Snapshot create API name validation. (#4023)
add d96f207f3a HDDS-7512. [snapshot] List Snapshot returns an empty list for a non-existent bucket (#3993)
add f77dfa6f59 HDDS-6962. [Snapshot] Background Service to delete irrelevant SST files in a snapshot. (#3883)
add e5d9c38f6f HDDS-7466. SnapshotDiff Client Code. (#3942)
add 2554ec26d7 HDDS-7508. [Snapshot] Finish TestOMSnapshotDAG, restore CompactionNode SnapshotID field for debugging (#3981)
add 1ee083178d HDDS-7484. Snapshot ID on followers should match the one on the OM Leader (#3985)
add 113072c16c HDDS-7583. Support efficient Snapdiff only until configured Snapshot history (#4036)
add 7f8838070e HDDS-7647. [snapshot] Add unit-testcases for Ozone Snapshot List API (#4090)
add 96bb4369c0 Revert "HDDS-7647. [snapshot] Add unit-testcases for Ozone Snapshot List API (#4090)" (#4102)
add 39bd6090bd Revert "HDDS-7583. Support efficient Snapdiff only until configured Snapshot history (#4036)" (#4103)
add 3d289e9050 HDDS-7647. [snapshot] Add unit-testcases for Ozone Snapshot List API (#4105)
add 9fe6d106ac HDDS-7524. Compaction DAG node pruning (#4045)
add 125eccfef2 HDDS-7607. [Snapshot] SnapshotDiff command lists unmodified file as modified (#4108)
add 0bcd697107 HDDS-7690. [Snapshot] Use SST file list output from compaction DAG as SnapshotDiff input (#4119)
add d8765436c2 Merge remote-tracking branch 'asf/master' into HDDS-6517-Snapshot
add 98e3a5033d Merge remote-tracking branch 'asf/master' into HDDS-6517-Snapshot
add 52fb8ba492 HDDS-7690. Addendum: [Snapshot] Use SST file list output from compaction DAG as SnapshotDiff input (#4168)
add 52103f273a HDDS-6984. [Snapshot] Prevent bucket from being deleted if snapshot exists (#4015)
add 0c25be8cdf HDDS-7702. [snapshot] Add unit-testcases for Ozone fs createSnapshot (#4122)
add 26006d4fcf HDDS-7765. [Snapshot] Handle OzoneManager#getKeyInfo with OmMetadataReader. (#4185)
add 01762a385d HDDS-7506. [Snapshot] Expose more snapshot metrics under OMMetrics (#4164)
add c88dd820e1 HDDS-7779. [snapshot] Add unit-testcases for Ozone SnapshotDiff.
add 3e31e047d9 HDDS-7768. [Snapshot] Add acceptance tests (#4171)
add dae9285af3 HDDS-7704. [snapshot] Add unit-testcases for snapshot restore (#4148)
add 40688f637a HDDS-7819. [Snapshot] Add unit-testcases for fs delete of bucket havng snapshots (#4198)
add ae83ac54e0 HDDS-7600. [Snapshot] Mark all new proto message fields as optional (#4120)
add aea5edfe99 HDDS-7845. [Snapshot] Wait for RocksDB checkpoint directory creation (#4214)
add 106b193c2a Merge remote-tracking branch 'asf/master' into HDDS-6517-Snapshot
add ccc814ee7f HDDS-6857. [Snapshot] Implement Snapshot Delete CLI and API (#4175)
add afa6030b90 HDDS-7861. [Snapshot] Delete keys from the source bucket to fix the flakiness of snapshot restore tests (#4230)
new f01cebe19b Merge remote-tracking branch 'asf/HDDS-6517-Snapshot'
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 32 +
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 7 +
.../common/src/main/resources/ozone-default.xml | 69 +
hadoop-hdds/framework/pom.xml | 10 +
.../hadoop/hdds/utils/BooleanTriFunction.java | 37 +
.../org/apache/hadoop/hdds/utils/db/DBProfile.java | 25 +
.../org/apache/hadoop/hdds/utils/db/DBStore.java | 6 +
.../hadoop/hdds/utils/db/DBStoreBuilder.java | 58 +-
.../hadoop/hdds/utils/db/RDBCheckpointManager.java | 67 +-
.../org/apache/hadoop/hdds/utils/db/RDBStore.java | 81 +-
.../apache/hadoop/hdds/utils/db/RocksDatabase.java | 132 ++
hadoop-hdds/pom.xml | 7 +
hadoop-hdds/rocksdb-checkpoint-differ/README.md | 18 +
.../dev-support/findbugsExcludeFile.xml | 0
hadoop-hdds/rocksdb-checkpoint-differ/pom.xml | 193 +++
.../ozone/rocksdb/util/ManagedSstFileReader.java | 128 ++
.../org/apache/ozone/rocksdb/util/RdbUtil.java | 61 +
.../apache/ozone/rocksdb/util/package-info.java | 22 +
.../org/apache/ozone/rocksdiff/CompactionNode.java | 80 ++
.../apache/ozone/rocksdiff/DifferSnapshotInfo.java | 63 +
.../apache/ozone/rocksdiff/RelationshipEdge.java | 30 +
.../ozone/rocksdiff/RocksDBCheckpointDiffer.java | 1345 ++++++++++++++++++++
.../org/apache/ozone/rocksdiff/RocksDiffUtils.java | 54 +
.../org/apache/ozone/rocksdiff}/package-info.java | 0
.../rocksdiff/TestRocksDBCheckpointDiffer.java | 1103 ++++++++++++++++
.../src/test/resources/log4j.properties | 0
.../apache/hadoop/ozone/client/ObjectStore.java | 43 +
.../apache/hadoop/ozone/client/OzoneBucket.java | 7 +-
.../apache/hadoop/ozone/client/OzoneSnapshot.java | 145 +++
.../ozone/client/protocol/ClientProtocol.java | 47 +
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 71 ++
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 27 +
.../org/apache/hadoop/ozone/audit/OMAction.java | 6 +-
.../apache/hadoop/ozone/om/IOmMetadataReader.java | 123 ++
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 14 +
.../hadoop/ozone/om/exceptions/OMException.java | 6 +-
.../apache/hadoop/ozone/om/helpers/OmKeyInfo.java | 3 +-
.../ozone/om/helpers/OmKeyLocationInfoGroup.java | 19 +
.../hadoop/ozone/om/helpers/SnapshotInfo.java | 515 ++++++++
.../hadoop/ozone/om/lock/IOzoneManagerLock.java | 98 ++
.../hadoop/ozone/om/lock/OmReadOnlyLock.java | 167 +++
.../hadoop/ozone/om/lock/OzoneManagerLock.java | 36 +-
.../hadoop/ozone/om/lock/OzoneManagerLockUtil.java | 12 +
.../ozone/om/protocol/OzoneManagerProtocol.java | 108 +-
...OzoneManagerProtocolClientSideTranslatorPB.java | 100 ++
.../hadoop/ozone/snapshot/SnapshotDiffReport.java | 233 ++++
.../apache/hadoop/ozone/snapshot/package-info.java | 22 +
.../hadoop/ozone/om/helpers/TestOmKeyInfo.java | 6 +-
.../ozone/om/helpers/TestOmSnapshotInfo.java | 147 +++
.../hadoop/ozone/om/lock/TestOzoneManagerLock.java | 9 +-
hadoop-ozone/dist/src/main/compose/ozone/test.sh | 2 +
hadoop-ozone/dist/src/main/license/bin/LICENSE.txt | 20 +-
hadoop-ozone/dist/src/main/license/jar-report.txt | 3 +
.../dist/src/main/smoketest/snapshot/basic.robot | 56 +
.../hadoop/fs/ozone/TestOzoneFsSnapshot.java | 323 +++++
.../hadoop/ozone/freon/TestOMSnapshotDAG.java | 269 ++++
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 42 +-
.../org/apache/hadoop/ozone/om/TestOmMetrics.java | 65 +
.../org/apache/hadoop/ozone/om/TestOmSnapshot.java | 662 ++++++++++
.../hadoop/ozone/om/TestOmSnapshotFileSystem.java | 581 +++++++++
.../om/snapshot/TestOzoneSnapshotRestore.java | 333 +++++
.../src/main/proto/OmClientProtocol.proto | 113 ++
.../apache/hadoop/ozone/om/OMMetadataManager.java | 16 +-
.../ozone/om/codec/OmDBSnapshotInfoCodec.java | 58 +
hadoop-ozone/ozone-manager/pom.xml | 4 +
.../apache/hadoop/ozone/om/BucketManagerImpl.java | 111 --
.../java/org/apache/hadoop/ozone/om/IOzoneAcl.java | 31 -
.../org/apache/hadoop/ozone/om/KeyManager.java | 6 +
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 244 +---
.../java/org/apache/hadoop/ozone/om/OMMetrics.java | 116 +-
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 197 ++-
.../apache/hadoop/ozone/om/OmMetadataReader.java | 583 +++++++++
.../hadoop/ozone/om/OmMetadataReaderMetrics.java | 49 +
.../org/apache/hadoop/ozone/om/OmSnapshot.java | 265 ++++
.../apache/hadoop/ozone/om/OmSnapshotManager.java | 271 ++++
.../apache/hadoop/ozone/om/OmSnapshotMetrics.java | 163 +++
.../org/apache/hadoop/ozone/om/OzoneAclUtils.java | 16 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 578 +++------
.../apache/hadoop/ozone/om/PrefixManagerImpl.java | 104 --
.../apache/hadoop/ozone/om/SnapshotChainInfo.java | 57 +
.../hadoop/ozone/om/SnapshotChainManager.java | 512 ++++++++
.../hadoop/ozone/om/SstFilteringService.java | 212 +++
.../apache/hadoop/ozone/om/VolumeManagerImpl.java | 111 --
.../hadoop/ozone/om/codec/OMDBDefinition.java | 13 +-
.../ozone/om/ratis/OzoneManagerDoubleBuffer.java | 431 ++++---
.../om/ratis/utils/OzoneManagerRatisUtils.java | 7 +-
.../hadoop/ozone/om/request/OMClientRequest.java | 17 +-
.../om/request/bucket/OMBucketDeleteRequest.java | 56 +
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 2 +-
.../request/snapshot/OMSnapshotCreateRequest.java | 190 +++
.../request/snapshot/OMSnapshotDeleteRequest.java | 227 ++++
.../ozone/om/request/snapshot/package-info.java | 23 +
.../om/request/validation/RequestValidations.java | 25 +-
.../om/request/validation/ValidatorRegistry.java | 7 +-
.../snapshot/OMSnapshotCreateResponse.java | 72 ++
.../snapshot/OMSnapshotDeleteResponse.java | 67 +
.../ozone/om/response/snapshot/package-info.java | 23 +
.../ozone/om/snapshot/SnapshotDiffManager.java | 390 ++++++
...OzoneManagerProtocolServerSideTranslatorPB.java | 95 +-
.../protocolPB/OzoneManagerRequestHandler.java | 40 +-
.../hadoop/ozone/om/TestOmMetadataManager.java | 219 +++-
.../hadoop/ozone/om/TestOmSnapshotManager.java | 128 ++
.../apache/hadoop/ozone/om/TestSnapshotChain.java | 306 +++++
.../apache/hadoop/ozone/om/TestSnapshotInfo.java | 102 ++
.../hadoop/ozone/om/TestSstFilteringService.java | 294 +++++
.../om/ratis/TestOzoneManagerDoubleBuffer.java | 216 ++++
.../ozone/om/request/OMRequestTestUtils.java | 86 ++
.../ozone/om/request/key/TestOMKeyRequest.java | 4 +
.../snapshot/TestOMSnapshotCreateRequest.java | 277 ++++
.../snapshot/TestOMSnapshotDeleteRequest.java | 331 +++++
.../ozone/om/request/snapshot/package-info.java | 23 +
.../snapshot/TestOMSnapshotCreateResponse.java | 122 ++
.../snapshot/TestOMSnapshotDeleteResponse.java | 125 ++
.../ozone/om/response/snapshot/package-info.java | 23 +
.../fs/ozone/BasicOzoneClientAdapterImpl.java | 10 +
.../hadoop/fs/ozone/BasicOzoneFileSystem.java | 10 +-
.../ozone/BasicRootedOzoneClientAdapterImpl.java | 9 +
.../fs/ozone/BasicRootedOzoneFileSystem.java | 9 +
.../apache/hadoop/fs/ozone/OzoneClientAdapter.java | 2 +
hadoop-ozone/pom.xml | 5 +
.../recon/recovery/ReconOmMetadataManagerImpl.java | 2 +-
.../hadoop/ozone/client/ClientProtocolStub.java | 28 +
.../shell/snapshot/CreateSnapshotHandler.java | 62 +
.../shell/snapshot/DeleteSnapshotHandler.java | 61 +
.../ozone/shell/snapshot/ListSnapshotHandler.java | 62 +
.../ozone/shell/snapshot/SnapshotCommands.java | 75 ++
.../ozone/shell/snapshot/SnapshotDiffHandler.java | 67 +
.../hadoop/ozone/shell/snapshot/package-info.java | 23 +
pom.xml | 9 +-
129 files changed, 14752 insertions(+), 1388 deletions(-)
create mode 100644 hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/BooleanTriFunction.java
create mode 100644 hadoop-hdds/rocksdb-checkpoint-differ/README.md
copy {hadoop-ozone/insight => hadoop-hdds/rocksdb-checkpoint-differ}/dev-support/findbugsExcludeFile.xml (100%)
create mode 100644 hadoop-hdds/rocksdb-checkpoint-differ/pom.xml
create mode 100644 hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdb/util/ManagedSstFileReader.java
create mode 100644 hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdb/util/RdbUtil.java
create mode 100644 hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdb/util/package-info.java
create mode 100644 hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdiff/CompactionNode.java
create mode 100644 hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdiff/DifferSnapshotInfo.java
create mode 100644 hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdiff/RelationshipEdge.java
create mode 100644 hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdiff/RocksDBCheckpointDiffer.java
create mode 100644 hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdiff/RocksDiffUtils.java
copy hadoop-hdds/{container-service/src/main/java/org/apache/hadoop/ozone => rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdiff}/package-info.java (100%)
create mode 100644 hadoop-hdds/rocksdb-checkpoint-differ/src/test/java/org/apache/ozone/rocksdiff/TestRocksDBCheckpointDiffer.java
copy {hadoop-ozone/common => hadoop-hdds/rocksdb-checkpoint-differ}/src/test/resources/log4j.properties (100%)
create mode 100644 hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneSnapshot.java
create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/IOmMetadataReader.java
create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/SnapshotInfo.java
create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/IOzoneManagerLock.java
create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/lock/OmReadOnlyLock.java
create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/snapshot/SnapshotDiffReport.java
create mode 100644 hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/snapshot/package-info.java
create mode 100644 hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestOmSnapshotInfo.java
create mode 100644 hadoop-ozone/dist/src/main/smoketest/snapshot/basic.robot
create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFsSnapshot.java
create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/freon/TestOMSnapshotDAG.java
create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshot.java
create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotFileSystem.java
create mode 100644 hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/snapshot/TestOzoneSnapshotRestore.java
create mode 100644 hadoop-ozone/interface-storage/src/main/java/org/apache/hadoop/ozone/om/codec/OmDBSnapshotInfoCodec.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataReader.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataReaderMetrics.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshot.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotMetrics.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotChainInfo.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SnapshotChainManager.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/SstFilteringService.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotCreateRequest.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotDeleteRequest.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/package-info.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/snapshot/OMSnapshotCreateResponse.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/snapshot/OMSnapshotDeleteResponse.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/snapshot/package-info.java
create mode 100644 hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmSnapshotManager.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestSnapshotChain.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestSnapshotInfo.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestSstFilteringService.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerDoubleBuffer.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/snapshot/TestOMSnapshotCreateRequest.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/snapshot/TestOMSnapshotDeleteRequest.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/snapshot/package-info.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/snapshot/TestOMSnapshotCreateResponse.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/snapshot/TestOMSnapshotDeleteResponse.java
create mode 100644 hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/snapshot/package-info.java
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/snapshot/CreateSnapshotHandler.java
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/snapshot/DeleteSnapshotHandler.java
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/snapshot/ListSnapshotHandler.java
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/snapshot/SnapshotCommands.java
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/snapshot/SnapshotDiffHandler.java
create mode 100644 hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/snapshot/package-info.java
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@ozone.apache.org
For additional commands, e-mail: commits-help@ozone.apache.org
[ozone] 01/01: Merge remote-tracking branch 'asf/HDDS-6517-Snapshot'
Posted by si...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
siyao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ozone.git
commit f01cebe19b62eaa00bb62d606900b83815fbf1f9
Merge: 1a796f1b54 afa6030b90
Author: Siyao Meng <50...@users.noreply.github.com>
AuthorDate: Wed Feb 1 21:18:43 2023 -0800
Merge remote-tracking branch 'asf/HDDS-6517-Snapshot'
Change-Id: I98122590b3b5c81d6d7afd24ae56a6b0d43eca5e
.../org/apache/hadoop/ozone/OzoneConfigKeys.java | 32 +
.../java/org/apache/hadoop/ozone/OzoneConsts.java | 7 +
.../common/src/main/resources/ozone-default.xml | 69 +
hadoop-hdds/framework/pom.xml | 10 +
.../hadoop/hdds/utils/BooleanTriFunction.java | 37 +
.../org/apache/hadoop/hdds/utils/db/DBProfile.java | 25 +
.../org/apache/hadoop/hdds/utils/db/DBStore.java | 6 +
.../hadoop/hdds/utils/db/DBStoreBuilder.java | 58 +-
.../hadoop/hdds/utils/db/RDBCheckpointManager.java | 67 +-
.../org/apache/hadoop/hdds/utils/db/RDBStore.java | 81 +-
.../apache/hadoop/hdds/utils/db/RocksDatabase.java | 132 ++
hadoop-hdds/pom.xml | 7 +
hadoop-hdds/rocksdb-checkpoint-differ/README.md | 18 +
.../dev-support/findbugsExcludeFile.xml | 19 +
hadoop-hdds/rocksdb-checkpoint-differ/pom.xml | 193 +++
.../ozone/rocksdb/util/ManagedSstFileReader.java | 128 ++
.../org/apache/ozone/rocksdb/util/RdbUtil.java | 61 +
.../apache/ozone/rocksdb/util/package-info.java | 22 +
.../org/apache/ozone/rocksdiff/CompactionNode.java | 80 ++
.../apache/ozone/rocksdiff/DifferSnapshotInfo.java | 63 +
.../apache/ozone/rocksdiff/RelationshipEdge.java | 30 +
.../ozone/rocksdiff/RocksDBCheckpointDiffer.java | 1345 ++++++++++++++++++++
.../org/apache/ozone/rocksdiff/RocksDiffUtils.java | 54 +
.../org/apache/ozone/rocksdiff/package-info.java | 23 +
.../rocksdiff/TestRocksDBCheckpointDiffer.java | 1103 ++++++++++++++++
.../src/test/resources/log4j.properties | 21 +
.../apache/hadoop/ozone/client/ObjectStore.java | 43 +
.../apache/hadoop/ozone/client/OzoneBucket.java | 7 +-
.../apache/hadoop/ozone/client/OzoneSnapshot.java | 145 +++
.../ozone/client/protocol/ClientProtocol.java | 47 +
.../apache/hadoop/ozone/client/rpc/RpcClient.java | 71 ++
.../main/java/org/apache/hadoop/ozone/OmUtils.java | 27 +
.../org/apache/hadoop/ozone/audit/OMAction.java | 6 +-
.../apache/hadoop/ozone/om/IOmMetadataReader.java | 123 ++
.../org/apache/hadoop/ozone/om/OMConfigKeys.java | 14 +
.../hadoop/ozone/om/exceptions/OMException.java | 6 +-
.../apache/hadoop/ozone/om/helpers/OmKeyInfo.java | 3 +-
.../ozone/om/helpers/OmKeyLocationInfoGroup.java | 19 +
.../hadoop/ozone/om/helpers/SnapshotInfo.java | 515 ++++++++
.../hadoop/ozone/om/lock/IOzoneManagerLock.java | 98 ++
.../hadoop/ozone/om/lock/OmReadOnlyLock.java | 167 +++
.../hadoop/ozone/om/lock/OzoneManagerLock.java | 36 +-
.../hadoop/ozone/om/lock/OzoneManagerLockUtil.java | 12 +
.../ozone/om/protocol/OzoneManagerProtocol.java | 108 +-
...OzoneManagerProtocolClientSideTranslatorPB.java | 100 ++
.../hadoop/ozone/snapshot/SnapshotDiffReport.java | 233 ++++
.../apache/hadoop/ozone/snapshot/package-info.java | 22 +
.../hadoop/ozone/om/helpers/TestOmKeyInfo.java | 6 +-
.../ozone/om/helpers/TestOmSnapshotInfo.java | 147 +++
.../hadoop/ozone/om/lock/TestOzoneManagerLock.java | 9 +-
hadoop-ozone/dist/src/main/compose/ozone/test.sh | 2 +
hadoop-ozone/dist/src/main/license/bin/LICENSE.txt | 20 +-
hadoop-ozone/dist/src/main/license/jar-report.txt | 3 +
.../dist/src/main/smoketest/snapshot/basic.robot | 56 +
.../hadoop/fs/ozone/TestOzoneFsSnapshot.java | 323 +++++
.../hadoop/ozone/freon/TestOMSnapshotDAG.java | 269 ++++
.../apache/hadoop/ozone/om/TestKeyManagerImpl.java | 42 +-
.../org/apache/hadoop/ozone/om/TestOmMetrics.java | 65 +
.../org/apache/hadoop/ozone/om/TestOmSnapshot.java | 662 ++++++++++
.../hadoop/ozone/om/TestOmSnapshotFileSystem.java | 581 +++++++++
.../om/snapshot/TestOzoneSnapshotRestore.java | 333 +++++
.../src/main/proto/OmClientProtocol.proto | 113 ++
.../apache/hadoop/ozone/om/OMMetadataManager.java | 16 +-
.../ozone/om/codec/OmDBSnapshotInfoCodec.java | 58 +
hadoop-ozone/ozone-manager/pom.xml | 4 +
.../apache/hadoop/ozone/om/BucketManagerImpl.java | 111 --
.../java/org/apache/hadoop/ozone/om/IOzoneAcl.java | 31 -
.../org/apache/hadoop/ozone/om/KeyManager.java | 6 +
.../org/apache/hadoop/ozone/om/KeyManagerImpl.java | 244 +---
.../java/org/apache/hadoop/ozone/om/OMMetrics.java | 116 +-
.../hadoop/ozone/om/OmMetadataManagerImpl.java | 197 ++-
.../apache/hadoop/ozone/om/OmMetadataReader.java | 583 +++++++++
.../hadoop/ozone/om/OmMetadataReaderMetrics.java | 49 +
.../org/apache/hadoop/ozone/om/OmSnapshot.java | 265 ++++
.../apache/hadoop/ozone/om/OmSnapshotManager.java | 271 ++++
.../apache/hadoop/ozone/om/OmSnapshotMetrics.java | 163 +++
.../org/apache/hadoop/ozone/om/OzoneAclUtils.java | 16 +-
.../org/apache/hadoop/ozone/om/OzoneManager.java | 578 +++------
.../apache/hadoop/ozone/om/PrefixManagerImpl.java | 104 --
.../apache/hadoop/ozone/om/SnapshotChainInfo.java | 57 +
.../hadoop/ozone/om/SnapshotChainManager.java | 512 ++++++++
.../hadoop/ozone/om/SstFilteringService.java | 212 +++
.../apache/hadoop/ozone/om/VolumeManagerImpl.java | 111 --
.../hadoop/ozone/om/codec/OMDBDefinition.java | 13 +-
.../ozone/om/ratis/OzoneManagerDoubleBuffer.java | 431 ++++---
.../om/ratis/utils/OzoneManagerRatisUtils.java | 7 +-
.../hadoop/ozone/om/request/OMClientRequest.java | 17 +-
.../om/request/bucket/OMBucketDeleteRequest.java | 56 +
.../hadoop/ozone/om/request/key/OMKeyRequest.java | 2 +-
.../request/snapshot/OMSnapshotCreateRequest.java | 190 +++
.../request/snapshot/OMSnapshotDeleteRequest.java | 227 ++++
.../ozone/om/request/snapshot/package-info.java | 23 +
.../om/request/validation/RequestValidations.java | 25 +-
.../om/request/validation/ValidatorRegistry.java | 7 +-
.../snapshot/OMSnapshotCreateResponse.java | 72 ++
.../snapshot/OMSnapshotDeleteResponse.java | 67 +
.../ozone/om/response/snapshot/package-info.java | 23 +
.../ozone/om/snapshot/SnapshotDiffManager.java | 390 ++++++
...OzoneManagerProtocolServerSideTranslatorPB.java | 95 +-
.../protocolPB/OzoneManagerRequestHandler.java | 40 +-
.../hadoop/ozone/om/TestOmMetadataManager.java | 219 +++-
.../hadoop/ozone/om/TestOmSnapshotManager.java | 128 ++
.../apache/hadoop/ozone/om/TestSnapshotChain.java | 306 +++++
.../apache/hadoop/ozone/om/TestSnapshotInfo.java | 102 ++
.../hadoop/ozone/om/TestSstFilteringService.java | 294 +++++
.../om/ratis/TestOzoneManagerDoubleBuffer.java | 216 ++++
.../ozone/om/request/OMRequestTestUtils.java | 86 ++
.../ozone/om/request/key/TestOMKeyRequest.java | 4 +
.../snapshot/TestOMSnapshotCreateRequest.java | 277 ++++
.../snapshot/TestOMSnapshotDeleteRequest.java | 331 +++++
.../ozone/om/request/snapshot/package-info.java | 23 +
.../snapshot/TestOMSnapshotCreateResponse.java | 122 ++
.../snapshot/TestOMSnapshotDeleteResponse.java | 125 ++
.../ozone/om/response/snapshot/package-info.java | 23 +
.../fs/ozone/BasicOzoneClientAdapterImpl.java | 10 +
.../hadoop/fs/ozone/BasicOzoneFileSystem.java | 10 +-
.../ozone/BasicRootedOzoneClientAdapterImpl.java | 9 +
.../fs/ozone/BasicRootedOzoneFileSystem.java | 9 +
.../apache/hadoop/fs/ozone/OzoneClientAdapter.java | 2 +
hadoop-ozone/pom.xml | 5 +
.../recon/recovery/ReconOmMetadataManagerImpl.java | 2 +-
.../hadoop/ozone/client/ClientProtocolStub.java | 28 +
.../shell/snapshot/CreateSnapshotHandler.java | 62 +
.../shell/snapshot/DeleteSnapshotHandler.java | 61 +
.../ozone/shell/snapshot/ListSnapshotHandler.java | 62 +
.../ozone/shell/snapshot/SnapshotCommands.java | 75 ++
.../ozone/shell/snapshot/SnapshotDiffHandler.java | 67 +
.../hadoop/ozone/shell/snapshot/package-info.java | 23 +
pom.xml | 9 +-
129 files changed, 14815 insertions(+), 1388 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@ozone.apache.org
For additional commands, e-mail: commits-help@ozone.apache.org