[jira] [Created] (KAFKA-20890) KRaft broker startup can fail to restore persisted dynamic broker configs when the local metadata snapshot is missing or stale

["Tamas Kornai (Jira)" <ji...@apache.org>]

Tamas Kornai created KAFKA-20890:
------------------------------------

             Summary: KRaft broker startup can fail to restore persisted dynamic broker configs when the local metadata snapshot is missing or stale
                 Key: KAFKA-20890
                 URL: https://issues.apache.org/jira/browse/KAFKA-20890
             Project: Kafka
          Issue Type: Bug
          Components: config, core, kraft
    Affects Versions: 4.3.0, 4.2.0, 4.1.0, 4.0.0, 3.9.2
            Reporter: Tamas Kornai


On KRaft broker startup, persisted dynamic broker configurations may not be available when runtime components and their Reconfigurable implementations are initialized. When the latest local metadata snapshot is missing or does not yet contain the relevant ConfigRecord, the broker starts from its static configuration and receives the persisted dynamic configuration later through metadata-log replay.

 

That replay is processed as a live reconfiguration. A configuration whose final persisted value is valid can therefore be rejected by validation that compares the requested value with the component's current runtime state. The broker remains on its static value even though the dynamic configuration store, and consequently DescribeConfigs, can report the persisted dynamic value.

 

This was originally observed with num.recovery.threads.per.data.dir, but the underlying problem is not specific to recovery-thread configuration. It can affect dynamic configurations whose validation depends on current runtime state, initialization order, or another dynamically supplied setting.

 

## Concrete production observation

 

The static broker value was:

num.recovery.threads.per.data.dir=1

 

A cluster-default dynamic configuration was rolled out in two safe steps:

unset -> 2 -> 4

 

The original rollout succeeded. Brokers logged both runtime transitions:

Resizing recovery thread pool size for each data dir from 1 to 2

Resizing recovery thread pool size for each data dir from 2 to 4

 

During a subsequent ZooKeeper-to-KRaft transition, affected brokers rebuilt their configuration as:

1 -> 4 -> 1

 

Their final runtime KafkaConfig value was 1, while the stored cluster-default dynamic configuration continued to report 4 and there were no per-broker overrides.

 

Later attempts to apply 4 were rejected because the active runtime value was still 1:

Dynamic thread count update validation failed for num.recovery.threads.per.data.dir=4,

value should not be greater than double the current value 1

 

## Expected behavior

 
 * A broker restart restores the final persisted dynamic configuration before dependent runtime components become active, regardless of whether the latest local metadata snapshot happens to contain that configuration.
 * - Startup restoration is not rejected merely because the persisted final value is more than a live-update limit away from the static value.
 * - If applying a dynamic configuration fails, configuration APIs do not report that rejected value as the broker's effective configuration.

## Actual behavior

 
 * Without a suitable local snapshot, the broker initializes affected components using the static value.
 * - Metadata-log replay later presents the persisted value as a live update.
 * - Current-state-dependent validation can reject that update.
 * - The component remains on the static runtime value.
 * - The dynamic configuration cache can nevertheless contain and report the rejected persisted value, so the reported effective configuration differs from the active runtime state.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)