[jira] [Assigned] (HDDS-17098) Write a previousSnapshotVersions summary in snapshot local data YAML so OM startup can stop reading before the SST file lists

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-17098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng reassigned HDDS-17098:
---------------------------------

    Assignee: Siyao Meng

> Write a previousSnapshotVersions summary in snapshot local data YAML so OM startup can stop reading before the SST file lists
> -----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17098
>                 URL: https://issues.apache.org/jira/browse/HDDS-17098
>             Project: Apache Ozone
>          Issue Type: Sub-task
>          Components: Ozone Manager, Snapshot
>            Reporter: Siyao Meng
>            Assignee: Siyao Meng
>            Priority: Major
>
> To build the snapshot version graph at startup, {{OmSnapshotLocalDataManager#init}} needs only {{snapshotId}}, {{previousSnapshotId}}, {{version}} and the {{previousSnapshotVersion}} of each version. With HDDS-17079 these are read from parser events and SST file entries are skipped, but the whole file is still read, because each {{previousSnapshotVersion}} sits inside {{versionSstFileInfos}} next to its SST file list, and there is no way to know that the last version has been seen until the end of the mapping.
> On a test cluster with about 19,350 snapshots the YAMLs total 126 to 260 GB per OM (median about 9 MB per file, almost all SST file entries). With HDDS-17078 and HDDS-17079 the YAML phase is disk bound: about 26 minutes to read 175.6 GB on HDD.
> Proposal: also write a derived top level map {{previousSnapshotVersions}} (version to previousSnapshotVersion). YAML keys are emitted in sorted order, so it and the snapshot ids and version all precede {{versionSstFileInfos}}. When the summary is present, {{loadWithoutSstFiles}} stops at the {{versionSstFileInfos}} key and reads only the first few KB of the file. Files without it are still streamed in full.
> Compatibility:
> * Released readers (2.1.x, 2.2.x) construct the document by key and ignore unknown keys, so they load new files. Verified by loading a new file with the 2.2.1 classes.
> * The summary is left out of the checksum computation, so the checksum is the same with or without it, and both old and new code verify old and new files.
> * An older OM rewriting a file drops the summary; the file then falls back to the full stream. A stale summary cannot be produced.
> * No layout feature is needed, since the change is additive and every released reader tolerates it.
> Tradeoff: for files with the summary, startup no longer parses the SST file lists at all, so malformed content there is not detected at startup and surfaces on the next full load of that snapshot (invalid SST values already behave this way after HDDS-17079). YAMLs are written atomically, so truncated files are not expected.
> Existing files keep the old layout until rewritten (snapshot create, purge, defrag, orphan version cleanup, previous snapshot change). With defrag off by default, most existing files on an upgraded cluster are not rewritten, so the gain applies to snapshots created after the upgrade until older ones age out. A one time rewrite of existing files is left for a follow-up.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org