You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "swamirishi (via GitHub)" <gi...@apache.org> on 2025/05/27 18:24:25 UTC
[PR] HDDS-13003. [Design Doc] Snapshot Compaction to reduce storage footprint [ozone]
swamirishi opened a new pull request, #8514:
URL: https://github.com/apache/ozone/pull/8514
## What changes were proposed in this pull request?
Design doc to improve snapshot scale.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-13003
## How was this patch tested?
No tests required
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13003. [Design Doc] Snapshot Compaction to reduce storage footprint [ozone]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #8514:
URL: https://github.com/apache/ozone/pull/8514#discussion_r2109927273
##########
hadoop-hdds/docs/content/feature/SnapshotCompaction.md:
##########
@@ -0,0 +1,87 @@
+# Improving Snapshot Scale:
+
+[HDDS-13003](https://issues.apache.org/jira/browse/HDDS-13003)
+
+# Problem Statement
+
+In Apache Ozone, snapshots currently take a checkpoint of the Active Object Store (AOS) RocksDB each time a snapshot is created and track the compaction of SST files over time. This model works efficiently when snapshots are short-lived, as they merely serve as hard links to the AOS RocksDB. However, over time, if an older snapshot persists while significant churn occurs in the AOS RocksDB (due to compactions and writes), the snapshot RocksDB may diverge significantly from both the AOS RocksDB and other snapshot RocksDB instances. This divergence increases storage requirements linearly with the number of snapshots.
+
+# Solution Proposal:
+
+The primary inefficiency in the current snapshotting mechanism stems from constant RocksDB compactions in AOS, which can cause a key, file, or directory entry to appear in multiple SST files. Ideally, each unique key, file, or directory entry should reside in only one SST file, eliminating redundant storage and mitigating the multiplier effect caused by snapshots. If implemented correctly, the total RocksDB size would be proportional to the total number of unique keys in the system rather than the number of snapshots.
+
+## Snapshot Compaction:
+
+Currently, automatic RocksDB compactions are disabled for snapshot RocksDB to preserve snapshot diff performance, preventing any form of compaction. However, snapshots can be compacted if the next snapshot in the chain is a checkpoint of the previous snapshot plus a diff stored in a separate SST file. The proposed approach involves rewriting snapshots iteratively from the beginning of the snapshot chain and restructuring them in a separate directory. P.S This has got nothing to do with compacting snapshot’s rocksdb, we are not going to enable rocksdb auto compaction on snapshot rocksdb.
Review Comment:
[nitpick] Consider rephrasing this note for clarity and formality; for example, clarify that the proposal is not to enable auto compaction on the snapshot RocksDB instance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13003. [Design Doc] Snapshot Defragmentation to reduce storage footprint [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #8514: HDDS-13003. [Design Doc] Snapshot Defragmentation to reduce storage footprint
URL: https://github.com/apache/ozone/pull/8514
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13003. [Design Doc] Snapshot Compaction to reduce storage footprint [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #8514:
URL: https://github.com/apache/ozone/pull/8514#discussion_r2226881572
##########
hadoop-hdds/docs/content/feature/SnapshotCompaction.md:
##########
@@ -0,0 +1,93 @@
+# Improving Snapshot Scale:
+
+[HDDS-13003](https://issues.apache.org/jira/browse/HDDS-13003)
+
+# Problem Statement
+
+In Apache Ozone, snapshots currently take a checkpoint of the Active Object Store (AOS) RocksDB each time a snapshot is created and track the compaction of SST files over time. This model works efficiently when snapshots are short-lived, as they merely serve as hard links to the AOS RocksDB. However, over time, if an older snapshot persists while significant churn occurs in the AOS RocksDB (due to compactions and writes), the snapshot RocksDB may diverge significantly from both the AOS RocksDB and other snapshot RocksDB instances. This divergence increases storage requirements linearly with the number of snapshots.
+
+# Solution Proposal:
+
+The primary inefficiency in the current snapshotting mechanism stems from constant RocksDB compactions in AOS, which can cause a key, file, or directory entry to appear in multiple SST files. Ideally, each unique key, file, or directory entry should reside in only one SST file, eliminating redundant storage and mitigating the multiplier effect caused by snapshots. If implemented correctly, the total RocksDB size would be proportional to the total number of unique keys in the system rather than the number of snapshots.
+
+## Snapshot Compaction:
+
+Currently, automatic RocksDB compactions are disabled for snapshot RocksDB to preserve snapshot diff performance, preventing any form of compaction. However, snapshots can be compacted if the next snapshot in the chain is a checkpoint of the previous snapshot plus a diff stored in a separate SST file. The proposed approach involves rewriting snapshots iteratively from the beginning of the snapshot chain and restructuring them in a separate directory. P.S This has got nothing to do with compacting snapshot’s rocksdb, we are not going to enable rocksdb auto compaction on snapshot rocksdb.
Review Comment:
"next snapshot in the chain" --> next active snapshot in the chain?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13003. [Design Doc] Snapshot Compaction to reduce storage footprint [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on code in PR #8514:
URL: https://github.com/apache/ozone/pull/8514#discussion_r2123575436
##########
hadoop-hdds/docs/content/feature/SnapshotCompaction.md:
##########
@@ -0,0 +1,87 @@
+# Improving Snapshot Scale:
+
+[HDDS-13003](https://issues.apache.org/jira/browse/HDDS-13003)
+
+# Problem Statement
+
+In Apache Ozone, snapshots currently take a checkpoint of the Active Object Store (AOS) RocksDB each time a snapshot is created and track the compaction of SST files over time. This model works efficiently when snapshots are short-lived, as they merely serve as hard links to the AOS RocksDB. However, over time, if an older snapshot persists while significant churn occurs in the AOS RocksDB (due to compactions and writes), the snapshot RocksDB may diverge significantly from both the AOS RocksDB and other snapshot RocksDB instances. This divergence increases storage requirements linearly with the number of snapshots.
+
+# Solution Proposal:
+
+The primary inefficiency in the current snapshotting mechanism stems from constant RocksDB compactions in AOS, which can cause a key, file, or directory entry to appear in multiple SST files. Ideally, each unique key, file, or directory entry should reside in only one SST file, eliminating redundant storage and mitigating the multiplier effect caused by snapshots. If implemented correctly, the total RocksDB size would be proportional to the total number of unique keys in the system rather than the number of snapshots.
+
+## Snapshot Compaction:
+
+Currently, automatic RocksDB compactions are disabled for snapshot RocksDB to preserve snapshot diff performance, preventing any form of compaction. However, snapshots can be compacted if the next snapshot in the chain is a checkpoint of the previous snapshot plus a diff stored in a separate SST file. The proposed approach involves rewriting snapshots iteratively from the beginning of the snapshot chain and restructuring them in a separate directory. P.S This has got nothing to do with compacting snapshot’s rocksdb, we are not going to enable rocksdb auto compaction on snapshot rocksdb.
+
+1. ### Introducing a last compaction time:
+
+ A new boolean flag (`needsCompaction`) and timestamp (`lastCompactionTime`) will be added to snapshot metadata. If absent, `needsCompaction` will default to `true`.
+ A new list of Map\<String, List\<Longs\>\> (`sstFiles`) also needs to be added to snapshot info; this would be storing the original list of sst files in the uncompacted copy of the snapshot corresponding to keyTable/fileTable/DirectoryTable.
+ Since this is not going to be consistent across all OMs this would have to be written to a local yaml file inside the snapshot directory and this can be maintained in the SnapshotChainManager in memory on startup. So all updates should not go via ratis.
+
+2. ### Snapshot Cache Lock for Read Prevention
+
+ A snapshot lock will be introduced in the snapshot cache to prevent reads on a specific snapshot during compaction. This ensures no active reads occur while replacing the underlying RocksDB instance.
+
+3. ### Directory Structure Changes
+
+ Snapshots currently reside in the `db.checkpoints` directory. The proposal introduces a `db.checkpoints.compacted` directory for compacted snapshots. The directory format should be as follows:
+
+| om.db-\<snapshot\_id\>.\<compaction\_timestamp\> |
+| :---- |
+
+4. ### Optimized Snapshot Diff Computation:
+
+To compute a snapshot diff:
+
+* If both snapshots are compacted, their compacted versions will be used. The diff b/w two compacted snapshot should be present in one sst file.
+* If the target snapshot is uncompacted & the source snapshot is compacted(other way is not possible as we always compact snapshots in order) and if the DAG has all the sst files corresponding to the uncompacted snapshot version of the compacted snapshot which would be captured as part of the snapshot metadata once a snapshot is compacted, then an efficient diff can be performed with the information present in the DAG.
+* Otherwise, a full diff will be computed between the compacted source and the uncompacted target snapshot.
+* Changes in the full diff logic is required to check inode ids of sst files and remove the common sst files b/w source and target snapshots.
+
+
+5. ### Snapshot Compaction Workflow
+
+ Snapshot compaction should only occur once the snapshot has undergone SST filtering. The following steps outline the process:
Review Comment:
Thanks @swamirishi for working on this. Does this mean the `SstFilteringService` would run it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13003. [Design Doc] Snapshot Compaction to reduce storage footprint [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #8514:
URL: https://github.com/apache/ozone/pull/8514#discussion_r2115098871
##########
hadoop-hdds/docs/content/feature/SnapshotCompaction.md:
##########
@@ -0,0 +1,87 @@
+# Improving Snapshot Scale:
Review Comment:
Need the head to fix the rat error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13003. [Design Doc] Snapshot Compaction to reduce storage footprint [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #8514:
URL: https://github.com/apache/ozone/pull/8514#discussion_r2121739706
##########
hadoop-hdds/docs/content/feature/SnapshotCompaction.md:
##########
@@ -0,0 +1,87 @@
+# Improving Snapshot Scale:
+
+[HDDS-13003](https://issues.apache.org/jira/browse/HDDS-13003)
+
+# Problem Statement
+
+In Apache Ozone, snapshots currently take a checkpoint of the Active Object Store (AOS) RocksDB each time a snapshot is created and track the compaction of SST files over time. This model works efficiently when snapshots are short-lived, as they merely serve as hard links to the AOS RocksDB. However, over time, if an older snapshot persists while significant churn occurs in the AOS RocksDB (due to compactions and writes), the snapshot RocksDB may diverge significantly from both the AOS RocksDB and other snapshot RocksDB instances. This divergence increases storage requirements linearly with the number of snapshots.
+
+# Solution Proposal:
+
+The primary inefficiency in the current snapshotting mechanism stems from constant RocksDB compactions in AOS, which can cause a key, file, or directory entry to appear in multiple SST files. Ideally, each unique key, file, or directory entry should reside in only one SST file, eliminating redundant storage and mitigating the multiplier effect caused by snapshots. If implemented correctly, the total RocksDB size would be proportional to the total number of unique keys in the system rather than the number of snapshots.
+
+## Snapshot Compaction:
+
+Currently, automatic RocksDB compactions are disabled for snapshot RocksDB to preserve snapshot diff performance, preventing any form of compaction. However, snapshots can be compacted if the next snapshot in the chain is a checkpoint of the previous snapshot plus a diff stored in a separate SST file. The proposed approach involves rewriting snapshots iteratively from the beginning of the snapshot chain and restructuring them in a separate directory. P.S This has got nothing to do with compacting snapshot’s rocksdb, we are not going to enable rocksdb auto compaction on snapshot rocksdb.
+
+1. ### Introducing a last compaction time:
+
+ A new boolean flag (`needsCompaction`) and timestamp (`lastCompactionTime`) will be added to snapshot metadata. If absent, `needsCompaction` will default to `true`.
+ A new list of Map\<String, List\<Longs\>\> (`sstFiles`) also needs to be added to snapshot info; this would be storing the original list of sst files in the uncompacted copy of the snapshot corresponding to keyTable/fileTable/DirectoryTable.
+ Since this is not going to be consistent across all OMs this would have to be written to a local yaml file inside the snapshot directory and this can be maintained in the SnapshotChainManager in memory on startup. So all updates should not go via ratis.
+
+2. ### Snapshot Cache Lock for Read Prevention
+
+ A snapshot lock will be introduced in the snapshot cache to prevent reads on a specific snapshot during compaction. This ensures no active reads occur while replacing the underlying RocksDB instance.
+
+3. ### Directory Structure Changes
+
+ Snapshots currently reside in the `db.checkpoints` directory. The proposal introduces a `db.checkpoints.compacted` directory for compacted snapshots. The directory format should be as follows:
Review Comment:
clarification
```suggestion
Snapshots currently reside in the `db.checkpoints` directory. The proposal introduces a `db.checkpoints.compacted` directory for compacted snapshots. And the OM DB checkpoint directory name format inside the `db.checkpoints.compacted` directory should be as follows:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13003. [Design Doc] Snapshot Compaction to reduce storage footprint [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on PR #8514:
URL: https://github.com/apache/ozone/pull/8514#issuecomment-2951103558
> I suggest adding a section that summarises how each table is handled during compaction. The information is scattered now, grouping it together might make the doc easier to read.
thanks @chungen0126 for reviewing the doc. I will update the doc shortly capturing this info
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13003. [Design Doc] Snapshot Compaction to reduce storage footprint [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #8514:
URL: https://github.com/apache/ozone/pull/8514#discussion_r2109915624
##########
hadoop-hdds/docs/content/feature/SnapshotCompaction.md:
##########
@@ -0,0 +1,87 @@
+# Improving Snapshot Scale:
Review Comment:
Need a header:
```suggestion
---
title: "Improve Snapshot Compaction Scale"
weight: 1
menu:
main:
parent: Features
summary: Reduce the disk usage occupied by Ozone Snapshot metadata.
---
<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
# Improving Snapshot Scale:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13003. [Design Doc] Snapshot Defragmentation to reduce storage footprint [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #8514:
URL: https://github.com/apache/ozone/pull/8514#discussion_r2270757627
##########
hadoop-hdds/docs/content/feature/SnapshotCompaction.md:
##########
@@ -0,0 +1,87 @@
+# Improving Snapshot Scale:
+
+[HDDS-13003](https://issues.apache.org/jira/browse/HDDS-13003)
+
+# Problem Statement
+
+In Apache Ozone, snapshots currently take a checkpoint of the Active Object Store (AOS) RocksDB each time a snapshot is created and track the compaction of SST files over time. This model works efficiently when snapshots are short-lived, as they merely serve as hard links to the AOS RocksDB. However, over time, if an older snapshot persists while significant churn occurs in the AOS RocksDB (due to compactions and writes), the snapshot RocksDB may diverge significantly from both the AOS RocksDB and other snapshot RocksDB instances. This divergence increases storage requirements linearly with the number of snapshots.
+
+# Solution Proposal:
+
+The primary inefficiency in the current snapshotting mechanism stems from constant RocksDB compactions in AOS, which can cause a key, file, or directory entry to appear in multiple SST files. Ideally, each unique key, file, or directory entry should reside in only one SST file, eliminating redundant storage and mitigating the multiplier effect caused by snapshots. If implemented correctly, the total RocksDB size would be proportional to the total number of unique keys in the system rather than the number of snapshots.
+
+## Snapshot Compaction:
+
+Currently, automatic RocksDB compactions are disabled for snapshot RocksDB to preserve snapshot diff performance, preventing any form of compaction. However, snapshots can be compacted if the next snapshot in the chain is a checkpoint of the previous snapshot plus a diff stored in a separate SST file. The proposed approach involves rewriting snapshots iteratively from the beginning of the snapshot chain and restructuring them in a separate directory. P.S This has got nothing to do with compacting snapshot’s rocksdb, we are not going to enable rocksdb auto compaction on snapshot rocksdb.
+
+1. ### Introducing a last compaction time:
+
+ A new boolean flag (`needsCompaction`) and timestamp (`lastCompactionTime`) will be added to snapshot metadata. If absent, `needsCompaction` will default to `true`.
+ A new list of Map\<String, List\<Longs\>\> (`sstFiles`) also needs to be added to snapshot info; this would be storing the original list of sst files in the uncompacted copy of the snapshot corresponding to keyTable/fileTable/DirectoryTable.
+ Since this is not going to be consistent across all OMs this would have to be written to a local yaml file inside the snapshot directory and this can be maintained in the SnapshotChainManager in memory on startup. So all updates should not go via ratis.
+
+2. ### Snapshot Cache Lock for Read Prevention
+
+ A snapshot lock will be introduced in the snapshot cache to prevent reads on a specific snapshot during compaction. This ensures no active reads occur while replacing the underlying RocksDB instance.
+
+3. ### Directory Structure Changes
+
+ Snapshots currently reside in the `db.checkpoints` directory. The proposal introduces a `db.checkpoints.compacted` directory for compacted snapshots. The directory format should be as follows:
Review Comment:
Actually, snapshots are under `db.snapshots`. e.g.:
```
/var/lib/hadoop-ozone/om/data/db.snapshots
/var/lib/hadoop-ozone/om/data/db.snapshots/diffState
/var/lib/hadoop-ozone/om/data/db.snapshots/diffState/compaction-sst-backup
/var/lib/hadoop-ozone/om/data/db.snapshots/diffState/snapDiff
/var/lib/hadoop-ozone/om/data/db.snapshots/diffState/compaction-log
/var/lib/hadoop-ozone/om/data/db.snapshots/checkpointState
```
Snapshots would be under `./db.snapshots/checkpointState`
##########
hadoop-hdds/docs/content/feature/SnapshotCompaction.md:
##########
@@ -0,0 +1,87 @@
+# Improving Snapshot Scale:
+
+[HDDS-13003](https://issues.apache.org/jira/browse/HDDS-13003)
+
+# Problem Statement
+
+In Apache Ozone, snapshots currently take a checkpoint of the Active Object Store (AOS) RocksDB each time a snapshot is created and track the compaction of SST files over time. This model works efficiently when snapshots are short-lived, as they merely serve as hard links to the AOS RocksDB. However, over time, if an older snapshot persists while significant churn occurs in the AOS RocksDB (due to compactions and writes), the snapshot RocksDB may diverge significantly from both the AOS RocksDB and other snapshot RocksDB instances. This divergence increases storage requirements linearly with the number of snapshots.
+
+# Solution Proposal:
+
+The primary inefficiency in the current snapshotting mechanism stems from constant RocksDB compactions in AOS, which can cause a key, file, or directory entry to appear in multiple SST files. Ideally, each unique key, file, or directory entry should reside in only one SST file, eliminating redundant storage and mitigating the multiplier effect caused by snapshots. If implemented correctly, the total RocksDB size would be proportional to the total number of unique keys in the system rather than the number of snapshots.
+
+## Snapshot Compaction:
+
+Currently, automatic RocksDB compactions are disabled for snapshot RocksDB to preserve snapshot diff performance, preventing any form of compaction. However, snapshots can be compacted if the next snapshot in the chain is a checkpoint of the previous snapshot plus a diff stored in a separate SST file. The proposed approach involves rewriting snapshots iteratively from the beginning of the snapshot chain and restructuring them in a separate directory. P.S This has got nothing to do with compacting snapshot’s rocksdb, we are not going to enable rocksdb auto compaction on snapshot rocksdb.
+
+1. ### Introducing a last compaction time:
+
+ A new boolean flag (`needsCompaction`) and timestamp (`lastCompactionTime`) will be added to snapshot metadata. If absent, `needsCompaction` will default to `true`.
+ A new list of Map\<String, List\<Longs\>\> (`sstFiles`) also needs to be added to snapshot info; this would be storing the original list of sst files in the uncompacted copy of the snapshot corresponding to keyTable/fileTable/DirectoryTable.
+ Since this is not going to be consistent across all OMs this would have to be written to a local yaml file inside the snapshot directory and this can be maintained in the SnapshotChainManager in memory on startup. So all updates should not go via ratis.
+
+2. ### Snapshot Cache Lock for Read Prevention
+
+ A snapshot lock will be introduced in the snapshot cache to prevent reads on a specific snapshot during compaction. This ensures no active reads occur while replacing the underlying RocksDB instance.
+
+3. ### Directory Structure Changes
+
+ Snapshots currently reside in the `db.checkpoints` directory. The proposal introduces a `db.checkpoints.compacted` directory for compacted snapshots. The directory format should be as follows:
Review Comment:
Actually, snapshots are under `db.snapshots`. e.g.:
```
/var/lib/hadoop-ozone/om/data/db.snapshots
/var/lib/hadoop-ozone/om/data/db.snapshots/diffState
/var/lib/hadoop-ozone/om/data/db.snapshots/diffState/compaction-sst-backup
/var/lib/hadoop-ozone/om/data/db.snapshots/diffState/snapDiff
/var/lib/hadoop-ozone/om/data/db.snapshots/diffState/compaction-log
/var/lib/hadoop-ozone/om/data/db.snapshots/checkpointState
```
Newly created snapshots would be under `./db.snapshots/checkpointState`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13003. [Design Doc] Snapshot Defragmentation to reduce storage footprint [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8514:
URL: https://github.com/apache/ozone/pull/8514#issuecomment-3519267626
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13003. [Design Doc] Snapshot Compaction to reduce storage footprint [ozone]
Posted by "chungen0126 (via GitHub)" <gi...@apache.org>.
chungen0126 commented on PR #8514:
URL: https://github.com/apache/ozone/pull/8514#issuecomment-2935167936
I suggest adding a section that summarises how each table is handled during compaction. The information is scattered now, grouping it together might make the doc easier to read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13003. [Design Doc] Snapshot Compaction to reduce storage footprint [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #8514:
URL: https://github.com/apache/ozone/pull/8514#discussion_r2127219797
##########
hadoop-hdds/docs/content/feature/SnapshotCompaction.md:
##########
@@ -0,0 +1,87 @@
+# Improving Snapshot Scale:
+
+[HDDS-13003](https://issues.apache.org/jira/browse/HDDS-13003)
+
+# Problem Statement
+
+In Apache Ozone, snapshots currently take a checkpoint of the Active Object Store (AOS) RocksDB each time a snapshot is created and track the compaction of SST files over time. This model works efficiently when snapshots are short-lived, as they merely serve as hard links to the AOS RocksDB. However, over time, if an older snapshot persists while significant churn occurs in the AOS RocksDB (due to compactions and writes), the snapshot RocksDB may diverge significantly from both the AOS RocksDB and other snapshot RocksDB instances. This divergence increases storage requirements linearly with the number of snapshots.
+
+# Solution Proposal:
+
+The primary inefficiency in the current snapshotting mechanism stems from constant RocksDB compactions in AOS, which can cause a key, file, or directory entry to appear in multiple SST files. Ideally, each unique key, file, or directory entry should reside in only one SST file, eliminating redundant storage and mitigating the multiplier effect caused by snapshots. If implemented correctly, the total RocksDB size would be proportional to the total number of unique keys in the system rather than the number of snapshots.
+
+## Snapshot Compaction:
+
+Currently, automatic RocksDB compactions are disabled for snapshot RocksDB to preserve snapshot diff performance, preventing any form of compaction. However, snapshots can be compacted if the next snapshot in the chain is a checkpoint of the previous snapshot plus a diff stored in a separate SST file. The proposed approach involves rewriting snapshots iteratively from the beginning of the snapshot chain and restructuring them in a separate directory. P.S This has got nothing to do with compacting snapshot’s rocksdb, we are not going to enable rocksdb auto compaction on snapshot rocksdb.
+
+1. ### Introducing a last compaction time:
+
+ A new boolean flag (`needsCompaction`) and timestamp (`lastCompactionTime`) will be added to snapshot metadata. If absent, `needsCompaction` will default to `true`.
+ A new list of Map\<String, List\<Longs\>\> (`sstFiles`) also needs to be added to snapshot info; this would be storing the original list of sst files in the uncompacted copy of the snapshot corresponding to keyTable/fileTable/DirectoryTable.
+ Since this is not going to be consistent across all OMs this would have to be written to a local yaml file inside the snapshot directory and this can be maintained in the SnapshotChainManager in memory on startup. So all updates should not go via ratis.
+
+2. ### Snapshot Cache Lock for Read Prevention
+
+ A snapshot lock will be introduced in the snapshot cache to prevent reads on a specific snapshot during compaction. This ensures no active reads occur while replacing the underlying RocksDB instance.
+
+3. ### Directory Structure Changes
+
+ Snapshots currently reside in the `db.checkpoints` directory. The proposal introduces a `db.checkpoints.compacted` directory for compacted snapshots. The directory format should be as follows:
+
+| om.db-\<snapshot\_id\>.\<compaction\_timestamp\> |
+| :---- |
+
+4. ### Optimized Snapshot Diff Computation:
+
+To compute a snapshot diff:
+
+* If both snapshots are compacted, their compacted versions will be used. The diff b/w two compacted snapshot should be present in one sst file.
+* If the target snapshot is uncompacted & the source snapshot is compacted(other way is not possible as we always compact snapshots in order) and if the DAG has all the sst files corresponding to the uncompacted snapshot version of the compacted snapshot which would be captured as part of the snapshot metadata once a snapshot is compacted, then an efficient diff can be performed with the information present in the DAG.
+* Otherwise, a full diff will be computed between the compacted source and the uncompacted target snapshot.
+* Changes in the full diff logic is required to check inode ids of sst files and remove the common sst files b/w source and target snapshots.
+
+
+5. ### Snapshot Compaction Workflow
+
+ Snapshot compaction should only occur once the snapshot has undergone SST filtering. The following steps outline the process:
+1. **Create a RocksDB checkpoint** of the path previous snapshot corresponding to the bucket in the chain (if it exists).
+2. **Acquire the `SNAPSHOT_GC_LOCK`** for the snapshot ID to prevent garbage collection during compaction.
+ 1. If there is no path previous snapshot then
+ 1. take a checkpoint of the same rocksdb instance remove keys that don’t correspond to the bucket from tables `keyTable`, `fileTable`, `directoryTable,deletedTable,deletedDirectoryTable` by running rocksdb delete range api. We can trigger a forced manual compaction on the rocksdb instance(This can be behind a flag wherein the process can just work with the checkpoint of the rocksdb if the flag is disabled and not perform manual compaction). This should be done if the snapshot has never been compacted before i.e. if `lastCompactionTime` is zero or null. Otherwise just update the `needsCompaction` to False.
+ 2. If path previous snapshot exists:
+ 1. **Compute the diff** between tables (`keyTable`, `fileTable`, `directoryTable`) of the checkpoint and the current snapshot using snapshot diff functionality.
+ 2. **Flush changed objects** into separate SST files using the SST file writer, categorizing them by table type.
+ 3. **Ingest these SST files** into the RocksDB checkpoint using the `ingestFile` API.
+3. Truncate `deletedTable,deletedDirectoryTable,snapshotRenamedTable etc. (All tables excepting keyTable/fileTable/directoryTable)` in checkpointed rocksdb and ingest the entire table from deletedTable and deletedDirectoryTable from the current snapshot rocksdb.
+4. **Acquire the snapshot cache lock** to prevent snapshot access during directory updates.
Review Comment:
Note: Grab write lock as implemented in #8474
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13003. [Design Doc] Snapshot Defragmentation to reduce storage footprint [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #8514:
URL: https://github.com/apache/ozone/pull/8514#issuecomment-3726428968
Revive
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13003. [Design Doc] Snapshot Compaction to reduce storage footprint [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #8514:
URL: https://github.com/apache/ozone/pull/8514#discussion_r2112279509
##########
hadoop-hdds/docs/content/feature/SnapshotCompaction.md:
##########
@@ -0,0 +1,87 @@
+# Improving Snapshot Scale:
+
+[HDDS-13003](https://issues.apache.org/jira/browse/HDDS-13003)
+
+# Problem Statement
+
+In Apache Ozone, snapshots currently take a checkpoint of the Active Object Store (AOS) RocksDB each time a snapshot is created and track the compaction of SST files over time. This model works efficiently when snapshots are short-lived, as they merely serve as hard links to the AOS RocksDB. However, over time, if an older snapshot persists while significant churn occurs in the AOS RocksDB (due to compactions and writes), the snapshot RocksDB may diverge significantly from both the AOS RocksDB and other snapshot RocksDB instances. This divergence increases storage requirements linearly with the number of snapshots.
+
+# Solution Proposal:
+
+The primary inefficiency in the current snapshotting mechanism stems from constant RocksDB compactions in AOS, which can cause a key, file, or directory entry to appear in multiple SST files. Ideally, each unique key, file, or directory entry should reside in only one SST file, eliminating redundant storage and mitigating the multiplier effect caused by snapshots. If implemented correctly, the total RocksDB size would be proportional to the total number of unique keys in the system rather than the number of snapshots.
+
+## Snapshot Compaction:
+
+Currently, automatic RocksDB compactions are disabled for snapshot RocksDB to preserve snapshot diff performance, preventing any form of compaction. However, snapshots can be compacted if the next snapshot in the chain is a checkpoint of the previous snapshot plus a diff stored in a separate SST file. The proposed approach involves rewriting snapshots iteratively from the beginning of the snapshot chain and restructuring them in a separate directory. P.S This has got nothing to do with compacting snapshot’s rocksdb, we are not going to enable rocksdb auto compaction on snapshot rocksdb.
+
+1. ### Introducing a last compaction time:
+
+ A new boolean flag (`needsCompaction`) and timestamp (`lastCompactionTime`) will be added to snapshot metadata. If absent, `needsCompaction` will default to `true`.
+ A new list of Map\<String, List\<Longs\>\> (`sstFiles`) also needs to be added to snapshot info; this would be storing the original list of sst files in the uncompacted copy of the snapshot corresponding to keyTable/fileTable/DirectoryTable.
+ Since this is not going to be consistent across all OMs this would have to be written to a local yaml file inside the snapshot directory and this can be maintained in the SnapshotChainManager in memory on startup. So all updates should not go via ratis.
+
+2. ### Snapshot Cache Lock for Read Prevention
+
+ A snapshot lock will be introduced in the snapshot cache to prevent reads on a specific snapshot during compaction. This ensures no active reads occur while replacing the underlying RocksDB instance.
Review Comment:
Maybe add an extra line stating this is expected to be near instantaneous.
##########
hadoop-hdds/docs/content/feature/SnapshotCompaction.md:
##########
@@ -0,0 +1,87 @@
+# Improving Snapshot Scale:
+
+[HDDS-13003](https://issues.apache.org/jira/browse/HDDS-13003)
+
+# Problem Statement
+
+In Apache Ozone, snapshots currently take a checkpoint of the Active Object Store (AOS) RocksDB each time a snapshot is created and track the compaction of SST files over time. This model works efficiently when snapshots are short-lived, as they merely serve as hard links to the AOS RocksDB. However, over time, if an older snapshot persists while significant churn occurs in the AOS RocksDB (due to compactions and writes), the snapshot RocksDB may diverge significantly from both the AOS RocksDB and other snapshot RocksDB instances. This divergence increases storage requirements linearly with the number of snapshots.
+
+# Solution Proposal:
+
+The primary inefficiency in the current snapshotting mechanism stems from constant RocksDB compactions in AOS, which can cause a key, file, or directory entry to appear in multiple SST files. Ideally, each unique key, file, or directory entry should reside in only one SST file, eliminating redundant storage and mitigating the multiplier effect caused by snapshots. If implemented correctly, the total RocksDB size would be proportional to the total number of unique keys in the system rather than the number of snapshots.
+
+## Snapshot Compaction:
+
+Currently, automatic RocksDB compactions are disabled for snapshot RocksDB to preserve snapshot diff performance, preventing any form of compaction. However, snapshots can be compacted if the next snapshot in the chain is a checkpoint of the previous snapshot plus a diff stored in a separate SST file. The proposed approach involves rewriting snapshots iteratively from the beginning of the snapshot chain and restructuring them in a separate directory. P.S This has got nothing to do with compacting snapshot’s rocksdb, we are not going to enable rocksdb auto compaction on snapshot rocksdb.
+
+1. ### Introducing a last compaction time:
+
+ A new boolean flag (`needsCompaction`) and timestamp (`lastCompactionTime`) will be added to snapshot metadata. If absent, `needsCompaction` will default to `true`.
Review Comment:
needsCompaction tells the system whether a snapshot is pending compaction (true) or if it is already compacted and up to date (false). This helps manage and automate the compaction workflow, ensuring snapshots are efficiently stored and maintained.
##########
hadoop-hdds/docs/content/feature/SnapshotCompaction.md:
##########
@@ -0,0 +1,87 @@
+# Improving Snapshot Scale:
+
+[HDDS-13003](https://issues.apache.org/jira/browse/HDDS-13003)
+
+# Problem Statement
+
+In Apache Ozone, snapshots currently take a checkpoint of the Active Object Store (AOS) RocksDB each time a snapshot is created and track the compaction of SST files over time. This model works efficiently when snapshots are short-lived, as they merely serve as hard links to the AOS RocksDB. However, over time, if an older snapshot persists while significant churn occurs in the AOS RocksDB (due to compactions and writes), the snapshot RocksDB may diverge significantly from both the AOS RocksDB and other snapshot RocksDB instances. This divergence increases storage requirements linearly with the number of snapshots.
+
+# Solution Proposal:
+
+The primary inefficiency in the current snapshotting mechanism stems from constant RocksDB compactions in AOS, which can cause a key, file, or directory entry to appear in multiple SST files. Ideally, each unique key, file, or directory entry should reside in only one SST file, eliminating redundant storage and mitigating the multiplier effect caused by snapshots. If implemented correctly, the total RocksDB size would be proportional to the total number of unique keys in the system rather than the number of snapshots.
+
+## Snapshot Compaction:
+
+Currently, automatic RocksDB compactions are disabled for snapshot RocksDB to preserve snapshot diff performance, preventing any form of compaction. However, snapshots can be compacted if the next snapshot in the chain is a checkpoint of the previous snapshot plus a diff stored in a separate SST file. The proposed approach involves rewriting snapshots iteratively from the beginning of the snapshot chain and restructuring them in a separate directory. P.S This has got nothing to do with compacting snapshot’s rocksdb, we are not going to enable rocksdb auto compaction on snapshot rocksdb.
+
+1. ### Introducing a last compaction time:
+
+ A new boolean flag (`needsCompaction`) and timestamp (`lastCompactionTime`) will be added to snapshot metadata. If absent, `needsCompaction` will default to `true`.
+ A new list of Map\<String, List\<Longs\>\> (`sstFiles`) also needs to be added to snapshot info; this would be storing the original list of sst files in the uncompacted copy of the snapshot corresponding to keyTable/fileTable/DirectoryTable.
+ Since this is not going to be consistent across all OMs this would have to be written to a local yaml file inside the snapshot directory and this can be maintained in the SnapshotChainManager in memory on startup. So all updates should not go via ratis.
+
+2. ### Snapshot Cache Lock for Read Prevention
+
+ A snapshot lock will be introduced in the snapshot cache to prevent reads on a specific snapshot during compaction. This ensures no active reads occur while replacing the underlying RocksDB instance.
+
+3. ### Directory Structure Changes
+
+ Snapshots currently reside in the `db.checkpoints` directory. The proposal introduces a `db.checkpoint.compacted` directory for compacted snapshots. The directory format should be as follows:
+
+| om.db-\<snapshot\_id\>.\<compaction\_timestamp\> |
+| :---- |
+
+4. ### Optimized Snapshot Diff Computation:
+
+To compute a snapshot diff:
+
+* If both snapshots are compacted, their compacted versions will be used. The diff b/w two compacted snapshot should be present in one sst file.
+* If the target snapshot is uncompacted & the source snapshot is compacted(other way is not possible as we always compact snapshots in order) and if the DAG has all the sst files corresponding to the uncompacted snapshot version of the compacted snapshot which would be captured as part of the snapshot metadata once a snapshot is compacted, then an efficient diff can be performed with the information present in the DAG.
+* Otherwise, a full diff will be computed between the compacted source and the uncompacted target snapshot.
+* Changes in the full diff logic is required to check inode ids of sst files and remove the common sst files b/w source and target snapshots.
+
+
+5. ### Snapshot Compaction Workflow
+
+ Snapshot compaction should only occur once the snapshot has undergone SST filtering. The following steps outline the process:
+1. **Create a RocksDB checkpoint** of the path previous snapshot corresponding to the bucket in the chain (if it exists). If a compacted copy already exists, update `compactionTime` and set `needsCompaction` to `false`. This checkpoint can be created in a tmp directory.
+2. **Acquire the `SNAPSHOT_GC_LOCK`** for the snapshot ID to prevent garbage collection during compaction.
+3. **Compute the diff** between tables (`keyTable`, `fileTable`, `directoryTable`) of the checkpoint and the current snapshot using snapshot diff functionality.
+4. **Flush changed objects** into separate SST files using the SST file writer, categorizing them by table type.
+5. **Ingest these SST files** into the RocksDB checkpoint using the `ingestFile` API.
+6. Truncate `deletedTable,deletedDirectoryTable,snapshotRenamedTable etc. (All tables excepting keyTable/fileTable/directoryTable)` in checkpointed rocksdb and ingest the entire table from deletedTable and deletedDirectoryTable from the current snapshot rocksdb.
+7. **Acquire the snapshot cache lock** to prevent snapshot access during directory updates.
+8. **Move the checkpoint directory** into `db.checkpoint.compacted` with the format:
+
+| om.db-\<snapshot\_id\>.\<snapshot\_compaction\_time\> |
+| :---- |
+
+9. **Update snapshot metadata**, setting `lastCompactionTime` and marking `needsCompaction = false` and set the next snapshot in the chain is marked for compaction. The `sstFiles` is set by creating Map\<String, List\<Long\>\> from the uncompacted version of the snapshot and this is only set once i.e. `lastCompactionTime` should be zero.
+10. **Delete old uncompacted/compacted snapshots**, ensuring unreferenced uncompacted/compacted snapshots are purged during OM startup(This is to handle jvm crash after viii).
+11. **Release the snapshot cache lock** on the snapshot id. Now the snapshot is ready to be used to read.
+
+
+
+6. ### Computing Changed Objects Between Snapshots
+
+ The following steps outline how to compute changed objects:
+1. **Determine delta SST files**:
+ * Retrieve from DAG if the snapshot was uncompacted previously and the previous snapshot has an uncompacted copy.
+ * Otherwise, compute delta SST files by comparing SST files in both compacted RocksDBs.
+2. **Initialize SST file writers** for `keyTable`, `directoryTable`, and `fileTable`.
+3. **Iterate SST files in parallel**, reading and merging keys to maintain sorted order.(Similar to the MinHeapIterator instead of iterating through multiple tables we would be iterating through multiple sst files concurrently).
+4. **Compare keys** between snapshots to determine changes and write updated objects if and only if they have changed into the SST file.
+ * If the object is present in the target snapshot then do an sstFileWriter.put().
+ * If the object is present in source snapshot but not present in target snapshot then we just have to write a tombstone entry by calling sstFileWriter.delete().
+5. **Ingest these SST files** into the checkpointed RocksDB.
+
+7. ### Handling Snapshot Purge
+
+ Upon snapshot deletion, the `needsCompaction` flag for the next snapshot in the chain is set to `true`, ensuring compaction propagates incrementally across the snapshot chain.
+
+# Conclusion
+
+This approach effectively reduces storage overhead while maintaining efficient snapshot retrieval and diff computation. The total storage would be in the order of total number of keys in the snapshots \+ AOS by reducing overall redundancy of the objects while also making the snapshot diff computation for even older snapshots more computationally efficient.
+
+
+
Review Comment:
```suggestion
# Snapshot Compaction Workflows (Visualization)
## 1. Snapshot Compaction Workflow
```mermaid
flowchart TD
A[Start: Uncompacted Snapshot Exists] --> B[Has SST Filtering Occurred?]
B -- No --> Z[Wait for SST Filtering]
B -- Yes --> C[Create RocksDB Checkpoint of Previous Snapshot]
C --> D{Compacted Copy Exists?}
D -- Yes --> E[Update compactionTime, set needsCompaction=false]
D -- No --> F[Create Checkpoint in Temp Directory]
E --> G[Acquire SNAPSHOT_GC_LOCK]
F --> G
G --> H[Compute Diff between Checkpoint & Current Snapshot]
H --> I[Flush Changed Objects into SST Files by table]
I --> J[Ingest SST Files into Checkpointed RocksDB]
J --> K[Truncate/Replace deletedTable, etc.]
K --> L[Acquire Snapshot Cache Lock]
L --> M[Move Checkpoint Dir to db.checkpoint.compacted]
M --> N[Update Snapshot Metadata: lastCompactionTime, needsCompaction=false, set next snapshot needsCompaction=true, set sstFiles]
N --> O[Delete Old Uncompacted/Compacted Snapshots]
O --> P[Release Snapshot Cache Lock]
P --> Q[Compacted Snapshot Ready]
```
---
## 2. Computing Changed Objects Between Snapshots
```mermaid
flowchart TD
A[Start: Need Diff Between Snapshots] --> B[Determine delta SST files]
B -- DAG Info available --> C[Retrieve from DAG]
B -- Otherwise --> D[Compute delta by comparing SST files in both RocksDBs]
C --> E[Initialize SST file writers: keyTable, directoryTable, fileTable]
D --> E
E --> F[Iterate SST files in parallel, merge keys: MinHeapIterator-like]
F --> G[Compare keys between snapshots]
G --> H{Object in Target?}
H -- Yes --> I[sstFileWriter.put]
H -- No --> J[sstFileWriter.delete tombstone]
I --> K[Ingest SST Files into Checkpointed RocksDB]
J --> K
```
---
## 3. Handling Snapshot Purge
```mermaid
flowchart TD
A[Snapshot Deletion Requested] --> B[Set needsCompaction=true for next snapshot in chain]
B --> C[Next snapshot will be compacted incrementally]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13003. [Design Doc] Snapshot Compaction to reduce storage footprint [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #8514:
URL: https://github.com/apache/ozone/pull/8514#issuecomment-3081391871
One potential race condition would be when the snapshot compaction process is ongoing, a new OM node bootstraps and tries to download the DBs.
In such case I believe we had existing lock in-place to prevent that but we need to double check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13003. [Design Doc] Snapshot Defragmentation to reduce storage footprint [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #8514:
URL: https://github.com/apache/ozone/pull/8514#discussion_r2270757627
##########
hadoop-hdds/docs/content/feature/SnapshotCompaction.md:
##########
@@ -0,0 +1,87 @@
+# Improving Snapshot Scale:
+
+[HDDS-13003](https://issues.apache.org/jira/browse/HDDS-13003)
+
+# Problem Statement
+
+In Apache Ozone, snapshots currently take a checkpoint of the Active Object Store (AOS) RocksDB each time a snapshot is created and track the compaction of SST files over time. This model works efficiently when snapshots are short-lived, as they merely serve as hard links to the AOS RocksDB. However, over time, if an older snapshot persists while significant churn occurs in the AOS RocksDB (due to compactions and writes), the snapshot RocksDB may diverge significantly from both the AOS RocksDB and other snapshot RocksDB instances. This divergence increases storage requirements linearly with the number of snapshots.
+
+# Solution Proposal:
+
+The primary inefficiency in the current snapshotting mechanism stems from constant RocksDB compactions in AOS, which can cause a key, file, or directory entry to appear in multiple SST files. Ideally, each unique key, file, or directory entry should reside in only one SST file, eliminating redundant storage and mitigating the multiplier effect caused by snapshots. If implemented correctly, the total RocksDB size would be proportional to the total number of unique keys in the system rather than the number of snapshots.
+
+## Snapshot Compaction:
+
+Currently, automatic RocksDB compactions are disabled for snapshot RocksDB to preserve snapshot diff performance, preventing any form of compaction. However, snapshots can be compacted if the next snapshot in the chain is a checkpoint of the previous snapshot plus a diff stored in a separate SST file. The proposed approach involves rewriting snapshots iteratively from the beginning of the snapshot chain and restructuring them in a separate directory. P.S This has got nothing to do with compacting snapshot’s rocksdb, we are not going to enable rocksdb auto compaction on snapshot rocksdb.
+
+1. ### Introducing a last compaction time:
+
+ A new boolean flag (`needsCompaction`) and timestamp (`lastCompactionTime`) will be added to snapshot metadata. If absent, `needsCompaction` will default to `true`.
+ A new list of Map\<String, List\<Longs\>\> (`sstFiles`) also needs to be added to snapshot info; this would be storing the original list of sst files in the uncompacted copy of the snapshot corresponding to keyTable/fileTable/DirectoryTable.
+ Since this is not going to be consistent across all OMs this would have to be written to a local yaml file inside the snapshot directory and this can be maintained in the SnapshotChainManager in memory on startup. So all updates should not go via ratis.
+
+2. ### Snapshot Cache Lock for Read Prevention
+
+ A snapshot lock will be introduced in the snapshot cache to prevent reads on a specific snapshot during compaction. This ensures no active reads occur while replacing the underlying RocksDB instance.
+
+3. ### Directory Structure Changes
+
+ Snapshots currently reside in the `db.checkpoints` directory. The proposal introduces a `db.checkpoints.compacted` directory for compacted snapshots. The directory format should be as follows:
Review Comment:
Actually, snapshots are under `db.snapshots`. e.g.:
```
/var/lib/hadoop-ozone/om/data/db.snapshots
/var/lib/hadoop-ozone/om/data/db.snapshots/diffState
/var/lib/hadoop-ozone/om/data/db.snapshots/diffState/compaction-sst-backup
/var/lib/hadoop-ozone/om/data/db.snapshots/diffState/snapDiff
/var/lib/hadoop-ozone/om/data/db.snapshots/diffState/compaction-log
/var/lib/hadoop-ozone/om/data/db.snapshots/checkpointState
```
Newly created FS snapshots would be under `./db.snapshots/checkpointState`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13003. [Design Doc] Snapshot Defragmentation to reduce storage footprint [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #8514:
URL: https://github.com/apache/ozone/pull/8514#discussion_r2270757627
##########
hadoop-hdds/docs/content/feature/SnapshotCompaction.md:
##########
@@ -0,0 +1,87 @@
+# Improving Snapshot Scale:
+
+[HDDS-13003](https://issues.apache.org/jira/browse/HDDS-13003)
+
+# Problem Statement
+
+In Apache Ozone, snapshots currently take a checkpoint of the Active Object Store (AOS) RocksDB each time a snapshot is created and track the compaction of SST files over time. This model works efficiently when snapshots are short-lived, as they merely serve as hard links to the AOS RocksDB. However, over time, if an older snapshot persists while significant churn occurs in the AOS RocksDB (due to compactions and writes), the snapshot RocksDB may diverge significantly from both the AOS RocksDB and other snapshot RocksDB instances. This divergence increases storage requirements linearly with the number of snapshots.
+
+# Solution Proposal:
+
+The primary inefficiency in the current snapshotting mechanism stems from constant RocksDB compactions in AOS, which can cause a key, file, or directory entry to appear in multiple SST files. Ideally, each unique key, file, or directory entry should reside in only one SST file, eliminating redundant storage and mitigating the multiplier effect caused by snapshots. If implemented correctly, the total RocksDB size would be proportional to the total number of unique keys in the system rather than the number of snapshots.
+
+## Snapshot Compaction:
+
+Currently, automatic RocksDB compactions are disabled for snapshot RocksDB to preserve snapshot diff performance, preventing any form of compaction. However, snapshots can be compacted if the next snapshot in the chain is a checkpoint of the previous snapshot plus a diff stored in a separate SST file. The proposed approach involves rewriting snapshots iteratively from the beginning of the snapshot chain and restructuring them in a separate directory. P.S This has got nothing to do with compacting snapshot’s rocksdb, we are not going to enable rocksdb auto compaction on snapshot rocksdb.
+
+1. ### Introducing a last compaction time:
+
+ A new boolean flag (`needsCompaction`) and timestamp (`lastCompactionTime`) will be added to snapshot metadata. If absent, `needsCompaction` will default to `true`.
+ A new list of Map\<String, List\<Longs\>\> (`sstFiles`) also needs to be added to snapshot info; this would be storing the original list of sst files in the uncompacted copy of the snapshot corresponding to keyTable/fileTable/DirectoryTable.
+ Since this is not going to be consistent across all OMs this would have to be written to a local yaml file inside the snapshot directory and this can be maintained in the SnapshotChainManager in memory on startup. So all updates should not go via ratis.
+
+2. ### Snapshot Cache Lock for Read Prevention
+
+ A snapshot lock will be introduced in the snapshot cache to prevent reads on a specific snapshot during compaction. This ensures no active reads occur while replacing the underlying RocksDB instance.
+
+3. ### Directory Structure Changes
+
+ Snapshots currently reside in the `db.checkpoints` directory. The proposal introduces a `db.checkpoints.compacted` directory for compacted snapshots. The directory format should be as follows:
Review Comment:
Actually, snapshots are under `db.snapshots`. e.g.:
```
/var/lib/hadoop-ozone/om/data/db.snapshots
/var/lib/hadoop-ozone/om/data/db.snapshots/diffState
/var/lib/hadoop-ozone/om/data/db.snapshots/diffState/compaction-sst-backup
/var/lib/hadoop-ozone/om/data/db.snapshots/diffState/snapDiff
/var/lib/hadoop-ozone/om/data/db.snapshots/diffState/compaction-log
/var/lib/hadoop-ozone/om/data/db.snapshots/checkpointState
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13003. [Design Doc] Snapshot Compaction to reduce storage footprint [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #8514:
URL: https://github.com/apache/ozone/pull/8514#discussion_r2114844327
##########
hadoop-hdds/docs/content/feature/SnapshotCompaction.md:
##########
@@ -0,0 +1,87 @@
+# Improving Snapshot Scale:
+
+[HDDS-13003](https://issues.apache.org/jira/browse/HDDS-13003)
+
+# Problem Statement
+
+In Apache Ozone, snapshots currently take a checkpoint of the Active Object Store (AOS) RocksDB each time a snapshot is created and track the compaction of SST files over time. This model works efficiently when snapshots are short-lived, as they merely serve as hard links to the AOS RocksDB. However, over time, if an older snapshot persists while significant churn occurs in the AOS RocksDB (due to compactions and writes), the snapshot RocksDB may diverge significantly from both the AOS RocksDB and other snapshot RocksDB instances. This divergence increases storage requirements linearly with the number of snapshots.
+
+# Solution Proposal:
+
+The primary inefficiency in the current snapshotting mechanism stems from constant RocksDB compactions in AOS, which can cause a key, file, or directory entry to appear in multiple SST files. Ideally, each unique key, file, or directory entry should reside in only one SST file, eliminating redundant storage and mitigating the multiplier effect caused by snapshots. If implemented correctly, the total RocksDB size would be proportional to the total number of unique keys in the system rather than the number of snapshots.
+
+## Snapshot Compaction:
+
+Currently, automatic RocksDB compactions are disabled for snapshot RocksDB to preserve snapshot diff performance, preventing any form of compaction. However, snapshots can be compacted if the next snapshot in the chain is a checkpoint of the previous snapshot plus a diff stored in a separate SST file. The proposed approach involves rewriting snapshots iteratively from the beginning of the snapshot chain and restructuring them in a separate directory. P.S This has got nothing to do with compacting snapshot’s rocksdb, we are not going to enable rocksdb auto compaction on snapshot rocksdb.
+
+1. ### Introducing a last compaction time:
+
+ A new boolean flag (`needsCompaction`) and timestamp (`lastCompactionTime`) will be added to snapshot metadata. If absent, `needsCompaction` will default to `true`.
+ A new list of Map\<String, List\<Longs\>\> (`sstFiles`) also needs to be added to snapshot info; this would be storing the original list of sst files in the uncompacted copy of the snapshot corresponding to keyTable/fileTable/DirectoryTable.
+ Since this is not going to be consistent across all OMs this would have to be written to a local yaml file inside the snapshot directory and this can be maintained in the SnapshotChainManager in memory on startup. So all updates should not go via ratis.
Review Comment:
If we end up using a YAML file, I'd recommend adding a checksum for the file to prevent from corruption.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13003. [Design Doc] Snapshot Defragmentation to reduce storage footprint [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on PR #8514:
URL: https://github.com/apache/ozone/pull/8514#issuecomment-3215231758
I am going to push an update this design doc. Mostly about the change of terms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13003. [Design Doc] Snapshot Defragmentation to reduce storage footprint [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8514:
URL: https://github.com/apache/ozone/pull/8514#issuecomment-3549966028
Thank you for your contribution. This PR is being closed due to inactivity. If needed, feel free to reopen it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-13003. [Design Doc] Snapshot Defragmentation to reduce storage footprint [ozone]
Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl merged PR #8514:
URL: https://github.com/apache/ozone/pull/8514
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org