[jira] [Updated] (HDDS-17078) OM startup takes hours with many snapshots because snapshot local data YAMLs are parsed repeatedly on one thread

["Siyao Meng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDDS-17078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siyao Meng updated HDDS-17078:
------------------------------
        Parent: HDDS-13003
    Issue Type: Sub-task  (was: Bug)

> OM startup takes hours with many snapshots because snapshot local data YAMLs are parsed repeatedly on one thread
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17078
>                 URL: https://issues.apache.org/jira/browse/HDDS-17078
>             Project: Apache Ozone
>          Issue Type: Sub-task
>          Components: Ozone Manager, Snapshot
>            Reporter: Siyao Meng
>            Assignee: Siyao Meng
>            Priority: Major
>
> h3. Problem
> On a cluster with about 19k snapshots, OM takes about 10 hours to open its RPC port after a restart. During that time the OM main thread is fully busy (100% of one core) in OmSnapshotLocalDataManager initialization:
> {code}
> HashMap.put <- SafeConstructor.processDuplicateKeys / flattenMapping
> OmSnapshotLocalDataYaml$SnapshotLocalDataConstructor$ConstructSstFileInfo.construct
>   <- ConstructVersionMeta.construct <- ConstructSnapshotLocalData.construct
> YamlSerializer.load
> OmSnapshotLocalDataManager.tryLoadSnapshotLocalData
> OmSnapshotLocalDataManager.addVersionNodeWithDependents
> OmSnapshotLocalDataManager.init
> OmSnapshotLocalDataManager.<init>
> OmSnapshotManager.<init>
> OzoneManager.instantiateServices
> {code}
> The whole delay falls between "Ozone filesystem snapshot feature is enabled" and the next OM startup log line. The rest of startup takes seconds. This affects 2.2.0 and current master.
> h3. Analysis
> OmSnapshotLocalDataManager#init lists all snapshot local data YAMLs, sorts them by file name and loads each one. For each file, addVersionNodeWithDependents walks the previous snapshot chain and loads every snapshot YAML not yet in the version graph. A snapshot reached through that walk is loaded again when the outer loop reaches its own file. Since the file names are UUIDs, the order is effectively random with respect to the chain, so almost every YAML is parsed twice, all on one thread.
> Measurements on the affected cluster:
> * 19,358 YAMLs, 164 GB in total (median 9 MB, max 17.7 MB). A median file holds one version with about 36.7k SstFileInfo entries.
> * Observed parse rate during startup: about 10 MB/s, 1.4 s per file.
> * About 2 x 164 GB at 10 MB/s comes to about 9 hours, which matches the observed startup time.
> h3. Proposed fix
> Parse every YAML exactly once in init, keeping only its version metadata (SnapshotVersionsMeta), and let the dependency walk reuse that metadata instead of reloading previous snapshots from disk. Run the parsing on a dedicated thread pool of min(cores, 8) threads; 8 matches the YamlSerializer Yaml pool size, so more threads would only wait. Handling of unreadable or mismatched YAMLs is unchanged.
> Expected improvement: about 2x from parsing once, times up to 8x from parallel parsing.
> h3. Testing
> Extended TestOmSnapshotLocalDataManager#testInitWithExistingYamlFiles to count YAML loads during init. It fails on the current code (3 loads for 2 files) and passes with the fix (2 loads).
> h3. Related
> The YAMLs themselves are much larger than expected because each snapshot lists tens of thousands of small SST files. That will be tracked separately.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org