[jira] [Resolved] (YUNIKORN-3347) Improve scheduler config checksum generation and checks

["Wilfred Spiegelenburg (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YUNIKORN-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wilfred Spiegelenburg resolved YUNIKORN-3347.
---------------------------------------------
    Fix Version/s: 1.10.0
       Resolution: Fixed

Configuration validation now makes sure that the checksum is checked and that the checksum value is returned.

On the core side, if the checksum has not changed skip the queue config update. Prevents doing a lot of work for no real update.

> Improve scheduler config checksum generation and checks
> -------------------------------------------------------
>
>                 Key: YUNIKORN-3347
>                 URL: https://issues.apache.org/jira/browse/YUNIKORN-3347
>             Project: Apache YuniKorn
>          Issue Type: Bug
>          Components: core - scheduler, shim - kubernetes
>            Reporter: Siddharth Krishna
>            Assignee: Siddharth Krishna
>            Priority: Major
>              Labels: pull-request-available, release-notes
>             Fix For: 1.10.0
>
>
> The scheduler configuration checksum is not handled robustly. Need to improve it as follows:
> * Fix checksum detection in GetConfigurationString: detect the checksum line at the start and the end of the serialised config, and stop scanning the entire config to locate it. The current implementation walks and rebuilds the whole string, which is inefficient for large configs (the ones large enough to be gzip-compressed).
> * When loading a config, override an incorrect or missing checksum in the in-memory config with the correct calculated value, and log the action taken so a bad checksum can be traced.
> * Skip processing a config update when the checksum has not changed (i.e. the config content is effectively unchanged even if the ConfigMap was updated for another reason). Verify this path handles a not yet set previous config.
> * Make the calculated checksum reproducible from the config object: stripping the checksum line from a serialised config must equal the config serialised with no checksum set, so consumers can recompute the checksum without re-running the load code.
> * Expose the checksum and the checksum-comparison result in the ValidateConfResponse returned by the /ws/v1/validate-conf REST endpoint.
> Note: This changes the calculated checksum value for existing serialised configs, causing a one time config reload on upgrade. This must be called out in the 1.9 release notes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: dev-help@yunikorn.apache.org