You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by "eubnara (via GitHub)" <gi...@apache.org> on 2026/06/05 12:27:35 UTC
[PR] AMBARI-26614: Config and config-group changes are slow to propagate to agents on large clusters [ambari]
eubnara opened a new pull request, #4135:
URL: https://github.com/apache/ambari/pull/4135
## What changes were proposed in this pull request?
Speeds up propagation of config / config-group changes to agents on large
clusters (AMBARI-26614). Previously every change recomputed the per-host agent
config for all cluster hosts, sequentially and redundantly, and a config-group
change touching only a few member hosts still pushed config to every host.
- Scope config-group create/update to the affected member hosts (union of
previous and new members). New `ConfigHelper.updateAgentConfigs(Cluster,
List<Long> hostIds)` overload; the cluster-wide
`updateAgentConfigs(Set<String>)` is kept as a wrapper.
- Cache `Cluster#getDesiredConfigs` per batch instead of recomputing per host.
- Parallelize per-host recomputation on the existing `ThreadPools` ForkJoinPool
(no new/leaked pool); wait for completion and surface per-host failures.
- Drop the unnecessary deep sorting of the agent payload (`ClusterConfigs`
SortedMap → Map; agents don't depend on key order).
- Look up config groups by host id instead of scanning by hostname.
- Remove the leftover config-name unescaping. AMBARI-23463 added an
escape/unescape pair for config names; AMBARI-23538 removed the escaping
half, leaving the unescaping half with nothing to undo. It still ran on every
key for every host on every push (wasted work on the hot path), and since
nothing escapes the keys anymore it corrupted any key containing a backslash
sequence. Output is unchanged for normal keys.
Cluster-wide config changes still target all hosts; the hash-based
"send only if changed" mechanism is unchanged.
## How was this patch tested?
Existing unit tests pass: `ConfigHelperTest`, `ConfigGroupResourceProviderTest`.
`ConfigHelperTest` was updated to stub `ServiceComponentHost#getHost()` for the
config-group-by-hostId lookup path. Manually verified on a large cluster that a
config-group change pushes config only to its member hosts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For additional commands, e-mail: dev-help@ambari.apache.org
Re: [PR] AMBARI-26614: Config and config-group changes are slow to propagate to agents on large clusters [ambari]
Posted by "eubnara (via GitHub)" <gi...@apache.org>.
eubnara commented on code in PR #4135:
URL: https://github.com/apache/ambari/pull/4135#discussion_r3381666463
##########
ambari-server/src/main/java/org/apache/ambari/server/agent/stomp/AgentConfigsHolder.java:
##########
@@ -84,9 +99,50 @@ public void updateData(Long clusterId, List<Long> hostIds) throws AmbariExceptio
}
}
- for (Long hostId : hostIds) {
- AgentConfigsUpdateEvent agentConfigsUpdateEvent = configHelper.getHostActualConfigs(hostId);
- updateData(agentConfigsUpdateEvent);
+ final List<Long> targetHostIds = hostIds;
+ // Resolve each host's full (all-cluster) config so a host belonging to more than one cluster does
+ // not lose the other clusters' configs when its cached event is replaced. The shared cache
+ // (concurrent for the parallel path) resolves each cluster's desired configs at most once.
+ Map<Long, Map<String, DesiredConfig>> cachedClustersDesiredConfigs = new ConcurrentHashMap<>();
+ // IMPORTANT - DO NOT MOVE THIS READ INTO THE PARALLEL BLOCK BELOW.
+ // Pre-resolve the changed cluster's desired configs here, on the calling (request) thread, while
+ // that thread's write transaction is still open. The per-host recompute below runs on ForkJoinPool
+ // workers, and ConfigHelper.getHostActualConfigsExcludeCluster resolves a cluster's desired configs
+ // lazily via: cachedClustersDesiredConfigs.computeIfAbsent(clusterId, id -> cl.getDesiredConfigs(false))
+ // A pool worker is a DIFFERENT thread with no inherited persistence context/transaction (the
+ // EntityManager/transaction is bound to the writing thread), so that lazy read sees only the last
+ // COMMITTED snapshot - never this request's still-uncommitted change - and pushes the PREVIOUS
+ // config value to agents. Because .get() keeps this transaction waiting for the workers, the worker
+ // read is always pre-commit, producing a deterministic off-by-one: each change lands on agents one
+ // change late, and the host stays stale until an ambari-server/agent restart re-resolves it.
+ // Seeding the cache here (read-your-writes on the request thread => fresh value) makes computeIfAbsent
+ // a no-op for this cluster, so the workers never call getDesiredConfigs themselves.
+ cachedClustersDesiredConfigs.put(clusterId, clusters.get().getCluster(clusterId).getDesiredConfigs(false));
Review Comment:
I fixed the previous mentioned bug here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For additional commands, e-mail: dev-help@ambari.apache.org
Re: [PR] AMBARI-26614: Config and config-group changes are slow to propagate to agents on large clusters [ambari]
Posted by "eubnara (via GitHub)" <gi...@apache.org>.
eubnara commented on PR #4135:
URL: https://github.com/apache/ambari/pull/4135#issuecomment-4658496271
I guess I found some bugs. Please delay merge this PR. Thanks for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For additional commands, e-mail: dev-help@ambari.apache.org
Re: [PR] AMBARI-26614: Config and config-group changes are slow to propagate to agents on large clusters [ambari]
Posted by "JiaLiangC (via GitHub)" <gi...@apache.org>.
JiaLiangC merged PR #4135:
URL: https://github.com/apache/ambari/pull/4135
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ambari.apache.org
For additional commands, e-mail: dev-help@ambari.apache.org