You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Siyao Meng (Jira)" <ji...@apache.org> on 2026/10/17 00:44:00 UTC

[jira] [Created] (HDDS-17079) Skip SST file lists when loading snapshot local data YAMLs for the snapshot version graph

Siyao Meng created HDDS-17079:
---------------------------------

             Summary: Skip SST file lists when loading snapshot local data YAMLs for the snapshot version graph
                 Key: HDDS-17079
                 URL: https://issues.apache.org/jira/browse/HDDS-17079
             Project: Apache Ozone
          Issue Type: Sub-task
          Components: Ozone Manager, Snapshot
            Reporter: Siyao Meng
            Assignee: Siyao Meng


h3. Problem
To build its snapshot version graph, OmSnapshotLocalDataManager only needs each YAML's snapshotId, previousSnapshotId, version and the previousSnapshotVersion of each version. Every load still builds all SstFileInfo entries, which make up almost the whole file, and then throws them away. With many snapshots this is most of the CPU cost of OM startup, even after HDDS-17078 (parse each YAML once at startup).

h3. Analysis
Timing one 9 MB YAML from a cluster with about 19k snapshots (one version, about 36.7k SstFileInfo entries):
* Full load: 0.153 s per file (59 MB/s)
* Load without SST file lists: 0.088 s per file (103 MB/s)

h3. Proposed fix
Add a load mode to OmSnapshotLocalDataYaml (YamlFactory(true)) that drops each version's sstFiles node before construction, so none of its entries are built. All other fields load as before. OmSnapshotLocalDataManager uses this mode only in init and in the previous snapshot walk. All other reads and every write still use the full serializer.

The serializer used in this mode refuses to save, so data loaded without SST file lists cannot overwrite a YAML.

h3. Compatibility
The on disk format is unchanged. New code reads YAMLs written by existing releases, and nothing new is written.

One behavior change: startup still parses the whole document, so files that are not valid YAML are still skipped as before. But invalid values inside SST file entries are no longer detected at startup; they show up on the next full load of that snapshot.

h3. Testing
Added TestOmSnapshotLocalDataYaml#testLoadWithoutSstFiles. A YAML written by the regular serializer loads in the new mode with the same snapshot ids, versions, previousSnapshotVersion values, checksum and transaction info, and with empty SST file lists.

Depends on HDDS-17078.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org