You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "sureshanaparti (via GitHub)" <gi...@apache.org> on 2026/06/04 06:23:27 UTC
[PR] Indirect agent connection improvements [cloudstack]
sureshanaparti opened a new pull request, #13345:
URL: https://github.com/apache/cloudstack/pull/13345
### Description
This PR improves the Indirect agent connection handling, has the following improvements.
- Enhances the Host connecting logic to avoid connecting storm (where Agent opens multiple sockets against Management Server).
- Implements HostConnectProcess task where Host upon connection checks whether lock is available, traces Host connecting progress, status and timeout.
- Introduces AgentConnectStatusCommand, where Host checks whether lock for the Host is available (i.e. "previous" connect process is finished).
- Implementes logic to check whether Management Server has lock against Host (exposed MySQL DB lock presence via API)
- Removes synchronization on Host disconnect process, double-disconnect logic in clustered Management Server environment, added early removal from ping map (in case of combination ping timeout delay + synchronized disconnect process the Agent Manager submits more disconnect requests)
- Introduces parameterized connection and status check timeouts
- Implements backoff algorithm abstraction - can be used either constant backoff timeout or exponential with jitter to wait between connection Host attempts to Management Server
- Implements ServerAttache to be used on the Agent side of communication (similar to Attache on Management Server side)
- Enhances/Adds logs significantly to Host Agent and Agent Manager logic to trace Host connecting and disconnecting process, including ids, names, context UUIDs and timings (how much time took overall initialization/deinitialization)
- Adds logs to communication between Management Servers (PDU requests)
- Adds DB indexes to improve search performance, uses IDEMPOTENT_ADD_INDEX for safer DB schema updates
<!--- Describe your changes in DETAIL - And how has behaviour functionally changed. -->
<!-- For new features, provide link to FS, dev ML discussion etc. -->
<!-- In case of bug fix, the expected and actual behaviours, steps to reproduce. -->
<!-- When "Fixes: #<id>" is specified, the issue/PR will automatically be closed when this PR gets merged -->
<!-- For addressing multiple issues/PRs, use multiple "Fixes: #<id>" -->
<!-- Fixes: # -->
<!--- ******************************************************************************* -->
<!--- NOTE: AUTOMATION USES THE DESCRIPTIONS TO SET LABELS AND PRODUCE DOCUMENTATION. -->
<!--- PLEASE PUT AN 'X' in only **ONE** box -->
<!--- ******************************************************************************* -->
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] Build/CI
- [ ] Test (unit or integration test code)
### Feature/Enhancement Scale or Bug Severity
#### Feature/Enhancement Scale
- [x] Major
- [ ] Minor
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [ ] Major
- [ ] Minor
- [ ] Trivial
### Screenshots (if appropriate):
### How Has This Been Tested?
Checked the indirect agents connections (and re-connections) with KVM hosts, SSVM & CPVM.
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
#### How did you try to break this feature and the system with this change?
<!-- see how your change affects other areas of the code, etc. -->
<!-- Please read the [CONTRIBUTING](https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md) document -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-4649394804
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-5912447914
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19355
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "nvazquez (via GitHub) via commits" <co...@cloudstack.apache.org>.
nvazquez commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-6746217845
Hi @sureshanaparti please advise if further fixes are needed to address the Copilot comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "nvazquez (via GitHub)" <gi...@apache.org>.
nvazquez commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-6040276052
Thanks @winterhazel, sure
@blueorangutan test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#discussion_r4144777505
##########
agent/src/main/java/com/cloud/agent/HostConnectProcess.java:
##########
@@ -0,0 +1,343 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent;
+
+import com.cloud.agent.api.AgentConnectStatusAnswer;
+import com.cloud.agent.api.AgentConnectStatusCommand;
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.Command;
+import com.cloud.agent.api.StartupAnswer;
+import com.cloud.agent.api.StartupCommand;
+import com.cloud.agent.properties.AgentProperties;
+import com.cloud.agent.properties.AgentPropertiesFileHandler;
+import com.cloud.agent.transport.Request;
+import com.cloud.exception.CloudException;
+import com.cloud.exception.OperationTimedoutException;
+import com.cloud.host.Status;
+import com.cloud.resource.ResourceStatusUpdater;
+import com.cloud.resource.ServerResource;
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.cloud.utils.nio.Link;
+import org.apache.cloudstack.threadcontext.ThreadContextCommandUtil;
+import org.apache.cloudstack.threadcontext.ThreadContextUtil;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.LogManager;
+
+import java.io.IOException;
+import java.nio.channels.ClosedChannelException;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Predicate;
+
+public class HostConnectProcess {
+ private static final Logger logger = LogManager.getLogger(HostConnectProcess.class);
+
+ public static final int DEFAULT_ASYNC_COMMAND_TIMEOUT_SEC =
+ AgentPropertiesFileHandler.getPropertyValue(AgentProperties.ASYNC_COMMAND_TIMEOUT_SEC);
+
+ public static final int DEFAULT_ASYNC_STARTUP_COMMAND_TIMEOUT_SEC =
+ AgentPropertiesFileHandler.getPropertyValue(AgentProperties.ASYNC_STARTUP_COMMAND_TIMEOUT_SEC);
+
+ static final long HOST_STATUS_CHECK_INITIAL_DELAY_SEC = 10;
+ private long hostStatusCheckDelaySec = AgentPropertiesFileHandler.getPropertyValue(AgentProperties.AGENT_HOST_STATUS_CHECK_DELAY_SEC);
+ private final AtomicReference<ScheduledFuture<?>> hostStatusFutureRef = new AtomicReference<>();
+ private final Agent agent;
+ private ScheduledExecutorService hostStatusExecutor;
+
+ public HostConnectProcess(Agent agent) {
+ this.agent = agent;
+ initExecutors();
+ }
+
+ private void initExecutors() {
+ stop();
+ var threadFactory = new NamedThreadFactory("Agent-" + HostStatusTask.class.getSimpleName());
+ hostStatusExecutor = Executors.newScheduledThreadPool(1, threadFactory);
+ }
+
+ /**
+ * Stops the whole connect process and cancels all scheduled asynchronous tasks.
+ * Returns {@link Boolean#TRUE} if {@link HostConnectProcess} was waiting for {@link StartupAnswer}.
+ */
+ public boolean stop() {
+ logger.debug("Stopping connect process. The process is active: {}", isInProgress());
+ stopHostStatusExecutor();
+ logger.debug("Stopped executor");
+ Optional<? extends ScheduledFuture<?>> hostStatusOpt = Optional.ofNullable(hostStatusFutureRef.getAndSet(null))
+ .filter(Predicate.not(ScheduledFuture::isCancelled));
+
+ hostStatusOpt.ifPresent(future -> future.cancel(true));
+ logger.debug("Cancelled future");
+
+ return hostStatusOpt.isPresent();
+ }
+
+ private void stopHostStatusExecutor() {
+ if (hostStatusExecutor != null) {
+ hostStatusExecutor.shutdownNow();
+ hostStatusExecutor = null;
+ }
+ }
+
+ public void scheduleConnectProcess(Link link, boolean connectionTransfer) {
+ logger.debug("Scheduling connect process for {}", link);
+ initExecutors();
+
+ var task = new HostStatusTask(link, connectionTransfer, agent, hostStatusFutureRef);
+ var future = hostStatusExecutor.scheduleWithFixedDelay(ThreadContextUtil.wrapThreadContext(task),
+ HOST_STATUS_CHECK_INITIAL_DELAY_SEC,
+ hostStatusCheckDelaySec, TimeUnit.SECONDS);
+ hostStatusFutureRef.set(future);
+ }
+
+ /**
+ * Returns {@link Boolean#TRUE} if {@link HostStatusTask} created and scheduled.
+ * That means there is already {@link Status#Connecting} process is running.
+ */
+ public boolean isInProgress() {
+ return Optional.ofNullable(hostStatusFutureRef.get())
+ .filter(Predicate.not(ScheduledFuture::isCancelled)).isPresent();
+ }
+
+ public void updateHostStatusCheckDelay(int newDelaySec) {
+ logger.info("Updating host status check delay from {} to {} seconds", hostStatusCheckDelaySec, newDelaySec);
+ this.hostStatusCheckDelaySec = newDelaySec;
+ }
+
+ /**
+ * Task waits for the Host to be available to connect to submit {@link StartupCommand}.
+ * Checks Host status on Management Server cluster and submit {@link StartupCommand} only if there is no lock and
+ * Host is not {@link Status#Connecting}.
+ */
+ public static class HostStatusTask implements Runnable, AsyncSend {
+ private final Set<Status> operationalStatuses = Set.of(Status.Connecting, Status.Up, Status.Rebalancing);
+
+ private final Link link;
+ private final boolean forceConnect;
+ private final Agent agent;
+ private final AtomicReference<? extends ScheduledFuture<?>> futureRef;
+
+ public HostStatusTask(Link link, boolean forceConnect, Agent agent,
+ AtomicReference<? extends ScheduledFuture<?>> futureRef) {
+ logger.debug("{} created", this.getClass().getSimpleName());
+ this.link = link;
+ this.forceConnect = forceConnect;
+ this.agent = agent;
+ this.futureRef = futureRef;
+ }
+
+ private void cancel() {
+ logger.debug("Cancelling future");
+ Optional.ofNullable(futureRef.get())
+ .filter(Predicate.not(ScheduledFuture::isCancelled))
+ .ifPresent(future -> future.cancel(true));
Review Comment:
`futureRef` is shared by all generations of `HostStatusTask`. If a new connection replaces an in-flight task, the cancelled old task can still return from its status request and call `cancel()`, which reads the newly installed future and cancels the replacement process. Keep the future associated with the task generation or cancel it only when the reference still points to that task.
##########
framework/cluster/src/main/java/com/cloud/cluster/ClusterManagerImpl.java:
##########
@@ -1250,7 +1253,18 @@ public ConfigKey<?>[] getConfigKeys() {
private boolean pingManagementNode(final ManagementServerHostVO mshost) {
final String targetIp = mshost.getServiceIP();
- if ("127.0.0.1".equals(targetIp) || "0.0.0.0".equals(targetIp)) {
+ final String targetHostName = mshost.getName();
+ final String currentHostName = NetUtils.getCanonicalHostName();
+
+ // Prefer hostname (CNAME support) but fall back to the non-null service IP when name is unset.
+ final String connectTarget = StringUtils.isNotBlank(targetHostName) ? targetHostName : targetIp;
+
+ // Check for self-detection using hostname comparison (for CNAME support)
+ if (currentHostName != null && currentHostName.equals(targetHostName)) {
+ logger.info("ping management node cluster service can not be performed on self (hostname match: {})", currentHostName);
+ return false;
Review Comment:
This hostname self-check runs while `checkNodeConflict` is validating a peer with the same hostname. For a live duplicate node that has the same hostname but a different service IP, `pingManagementNode` returns false here instead of probing the peer, so `handleConflict` treats the active conflict as stale and allows startup. Self-detection should use the local management-server record/id (or otherwise distinguish the peer IP), not hostname equality alone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#discussion_r4398568028
##########
engine/orchestration/src/main/java/com/cloud/agent/manager/ClusteredAgentManagerImpl.java:
##########
@@ -510,14 +542,9 @@ public SocketChannel connectToPeer(final String peerName, final SocketChannel pr
logger.info("Unable to find peer: {}", peerName);
return null;
}
- final String ip = ms.getServiceIP();
- InetAddress addr;
- int port = Port.value();
- try {
- addr = InetAddress.getByName(ip);
- } catch (final UnknownHostException e) {
- throw new CloudRuntimeException("Unable to resolve " + ip);
- }
+ final String hostName = getHostIdentifier(ms);
+ final int port = Port.value();
+ final InetAddress addr = resolveManagementServerAddress(ms);
Review Comment:
`getHostIdentifier` always prefers `ManagementServerHost.getName()`, but `ClusterManagerImpl` stores that field as the node's canonical hostname even when `management.server.address` is configured with IPs. In an IP-configured deployment without DNS for that hostname, this replaces the previously working `serviceIP` and makes clustered peer connections fail. Select the identifier according to the configured address format (or fall back to `serviceIP`) rather than unconditionally preferring the persisted name.
##########
core/src/main/java/org/apache/cloudstack/threadcontext/ThreadContextCommandUtil.java:
##########
@@ -0,0 +1,74 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.threadcontext;
+
+import com.cloud.agent.api.Command;
+import com.cloud.utils.StringUtils;
+import org.apache.logging.log4j.ThreadContext;
+
+/**
+ * Utility class for Command-specific MDC operations.
+ * This class handles propagation of MDC values to and from Command objects.
+ *
+ * @author mprokopchuk
+ */
+public class ThreadContextCommandUtil {
+
+ /**
+ * Propagate UUID and log context ID from Command trace context to MDC.
+ * A key absent from the command is removed from MDC so that a worker thread does not
+ * carry over the context of a previously processed command.
+ *
+ * @param cmd the command containing trace context parameters
+ */
+ public static void propagateContextFromCommand(Command cmd) {
+ if (cmd == null) {
+ return;
+ }
+ putOrRemove(ThreadContextUtil.MDC_LOG_CONTEXT_ID_KEY, cmd.getTraceContextParam(ThreadContextUtil.CONTEXT_LOG_ID_KEY));
+ putOrRemove(ThreadContextUtil.MDC_UUID_KEY, cmd.getTraceContextParam(ThreadContextUtil.CONTEXT_UUID_KEY));
+ }
+
+ private static void putOrRemove(String mdcKey, String value) {
+ if (StringUtils.isNotEmpty(value)) {
+ ThreadContext.put(mdcKey, value);
+ } else {
+ ThreadContext.remove(mdcKey);
+ }
+ }
+
+ /**
+ * Set UUID and log context ID in Command trace context from current MDC values.
+ * MDC values are authoritative for the current thread; any pre-existing trace context
+ * on the command is intentionally overwritten.
+ *
+ * @param cmd the command to set trace context parameters on
+ */
+ public static void setContextInCommand(Command cmd) {
+ if (cmd != null) {
+ String logContextId = (String) ThreadContext.get(ThreadContextUtil.MDC_LOG_CONTEXT_ID_KEY);
+ if (StringUtils.isNotEmpty(logContextId)) {
+ cmd.setTraceContextParam(ThreadContextUtil.CONTEXT_LOG_ID_KEY, logContextId);
+ }
+
+ String uuid = (String) ThreadContext.get(ThreadContextUtil.MDC_UUID_KEY);
+ if (StringUtils.isNotEmpty(uuid)) {
+ cmd.setTraceContextParam(ThreadContextUtil.CONTEXT_UUID_KEY, uuid);
Review Comment:
This method claims MDC values are authoritative and that existing trace context is overwritten, but it only writes keys when the current MDC value is non-empty. Reusing a `Command` that already carries a UUID or log ID on a thread without that value leaves the old trace context attached and propagates it to the next operation. Clear the command entries when the corresponding MDC value is absent, rather than leaving the previous value in place.
##########
server/src/main/java/org/apache/cloudstack/agent/lb/IndirectAgentLBServiceImpl.java:
##########
@@ -595,16 +626,19 @@ protected void runInContext() {
msList = getManagementServerList(hostId, dcId, orderedHostIdList, lbAlgorithm);
}
+ // ask Host to reconnect to another Management Server
final MigrateAgentConnectionCommand cmd = new MigrateAgentConnectionCommand(msList, avoidMsList, lbAlgorithm, lbCheckInterval);
- cmd.setWait(60);
+ cmd.setWait(IndirectAgentLBMigrateCommandWait.value());
final Answer answer = agentManager.easySend(hostId, cmd); //may not receive answer when the agent disconnects immediately and try reconnecting to other ms host
if (answer == null) {
logger.warn(String.format("Got empty answer while initiating migration of agent connection for host agent ID: %d", hostId));
} else if (!answer.getResult()) {
Review Comment:
The null-answer branch immediately above records no failure, while this new `failedHostIds` tracking only covers unsuccessful answers and exceptions. A migration with no response is therefore omitted from the final failure warning. Add the host to `failedHostIds` in the null-answer branch before logging it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-6468497099
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19798
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-6276044630
This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#discussion_r3354005739
##########
engine/orchestration/src/main/java/com/cloud/agent/manager/ClusteredAgentManagerImpl.java:
##########
@@ -285,8 +290,9 @@ protected AgentAttache createAttache(final HostVO host) {
_agents.put(host.getId(), attache);
}
if (old != null) {
- logger.debug("Remove stale agent attache from current management server");
- removeAgent(old, Status.Removed);
+ logger.debug("Remove stale agent attache from current management server {}", _nodeId);
+ // just remove agent but do not deinitialize
+ removeAgent(old.getId(), attache);
}
Review Comment:
Bug: this call removes the newly created forwarding attache from the _agents map. At this point the map already contains `attache`, so `removeAgent(old.getId(), attache)` will `remove(hostId)` and return without restoring it, leaving no attache registered for the host on this node.
##########
engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java:
##########
@@ -1376,62 +1763,129 @@ protected AgentAttache createAttacheForConnect(final HostVO host, final Link lin
return attache;
}
- private AgentAttache sendReadyAndGetAttache(HostVO host, ReadyCommand ready, Link link, StartupCommand[] startupCmds) throws ConnectionException {
+ private AgentAttache sendReadyAndGetAttache(HostVO host, ReadyCommand ready, Link link, StartupCommand[] startup) throws ConnectionException {
AgentAttache attache;
GlobalLock joinLock = getHostJoinLock(host.getId());
try {
- if (!joinLock.lock(60)) {
- throw new ConnectionException(true, String.format("Unable to acquire lock on host %s, to process agent connection", host));
+ long processStart = System.currentTimeMillis();
+ if (joinLock.lock(getTimeoutSec())) {
+ logProcessingStart(host, joinLock);
+ try {
+ updateReadyCommandWithMSList(host, ready, startup);
+ attache = createAndNotifyAttache(host, link, startup);
+ } finally {
+ logProcessingFinish(host, joinLock, processStart);
+ joinLock.unlock();
+ }
+ } else {
+ throw createLockAcquisitionException(host, joinLock, processStart);
}
-
- logger.debug("Acquired lock on host {}, to process agent connection", host);
- attache = connectHostAgent(host, ready, link, startupCmds, joinLock);
} finally {
joinLock.releaseRef();
}
return attache;
}
- private AgentAttache connectHostAgent(HostVO host, ReadyCommand ready, Link link, StartupCommand[] startupCmds, GlobalLock joinLock) throws ConnectionException {
- AgentAttache attache;
- try {
- final List<String> agentMSHostList = new ArrayList<>();
- String lbAlgorithm = null;
- if (startupCmds != null && startupCmds.length > 0) {
- final String agentMSHosts = startupCmds[0].getMsHostList();
- if (StringUtils.isNotEmpty(agentMSHosts)) {
- String[] msHosts = agentMSHosts.split("@");
- if (msHosts.length > 1) {
- lbAlgorithm = msHosts[1];
- }
- agentMSHostList.addAll(Arrays.asList(msHosts[0].split(",")));
+ private void logProcessingStart(HostVO host, GlobalLock joinLock) {
+ StringBuilder msgBuilder = getSummaryMsgBuilder("Processing Host connection started",
+ EventTypes.EVENT_HOST_RECONNECT, host.getUuid(), host.getName(), joinLock.getName(), null, null, null);
+ logger.info(msgBuilder.toString());
+ }
+
+ private void logProcessingFinish(HostVO host, GlobalLock joinLock, long processStart) {
+ long processFinish = System.currentTimeMillis() - processStart;
+ StringBuilder msgBuilder = getSummaryMsgBuilder("Processing Host connection finished",
+ EventTypes.EVENT_HOST_RECONNECT, host.getUuid(), host.getName(), joinLock.getName(), null, null,
+ processFinish);
+ logger.fatal(msgBuilder.toString());
+ }
+
+ private void updateReadyCommandWithMSList(HostVO host, ReadyCommand ready, StartupCommand[] startup) {
+ List<String> agentMSHostList = new ArrayList<>();
+ String lbAlgorithm = null;
+
+ if (startup != null) {
+ String agentMSHosts = startup[0].getMsHostList();
+ if (StringUtils.isNotEmpty(agentMSHosts)) {
+ String[] msHosts = agentMSHosts.split("@");
Review Comment:
Potential AIOOBE: `startup[0]` is accessed when `startup != null` but without checking `startup.length > 0`. An empty startup array would crash the connect flow.
##########
engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java:
##########
@@ -1544,24 +2008,26 @@ protected void runInContext() {
}
protected void connectAgent(final Link link, final Command[] cmds, final Request request) {
- // send startupanswer to agent in the very beginning, so agent can move on without waiting for the answer for an undetermined time, if we put this logic into another
- // thread pool.
- final StartupAnswer[] answers = new StartupAnswer[cmds.length];
+ // send startup answer to agent in the very beginning, so agent can move on without waiting for the answer for an undetermined time,
+ // if we put this logic into another thread pool.
+ Map<String, String> backoffConfiguration = ConfigKeyUtil.toMap(BackoffConfiguration.value());
+ StartupAnswer[] answers = new StartupAnswer[cmds.length];
Command cmd;
for (int i = 0; i < cmds.length; i++) {
cmd = cmds[i];
- if (cmd instanceof StartupRoutingCommand || cmd instanceof StartupProxyCommand || cmd instanceof StartupSecondaryStorageCommand ||
- cmd instanceof StartupStorageCommand) {
- answers[i] = new StartupAnswer((StartupCommand) cmds[i], 0, "", "", mgmtServiceConf.getPingInterval());
- break;
+ if (cmd instanceof StartupRoutingCommand || cmd instanceof StartupProxyCommand || cmd instanceof StartupSecondaryStorageCommand
+ || cmd instanceof StartupStorageCommand) {
+ StartupAnswer answer = new StartupAnswer((StartupCommand) cmds[i], 0, "", "", mgmtServiceConf.getPingInterval());
+ answer.setParams(backoffConfiguration);
+ answer.setAgentHostStatusCheckDelaySec(AgentHostStatusCheckDelay.value());
+ answers[i] = answer;
}
}
- Response response;
- response = new Response(request, answers[0], _nodeId, -1);
+ Response response = new Response(request, answers[0], _nodeId, -1);
Review Comment:
Possible NPE: the loop fills `answers[i]`, but the response always uses `answers[0]`. If the first command isn't a Startup* command, `answers[0]` stays null and `new Response(...)` will fail. Even when it is, the loop should stop once the first startup answer is prepared.
##########
framework/cluster/src/main/java/com/cloud/cluster/ClusterServiceServletImpl.java:
##########
@@ -137,6 +140,12 @@ public boolean ping(final String callingPeer) throws RemoteException {
private String executePostMethod(final CloseableHttpClient client, final HttpPost method) {
String result = null;
+ String request = null;
+ try {
+ request = EntityUtils.toString(method.getEntity(), Charset.defaultCharset());
+ } catch (Exception e) {
+ logger.warn("Failed to retrieve request entity for POST {}", serviceUrl, e);
+ }
Review Comment:
`executePostMethod` always materializes the request entity into a String, even when debug logging is disabled. This adds overhead and also risks logging sensitive internal payloads (e.g. gsonPackage) when debug is enabled. Consider only building/logging the request string when needed, and redacting large/sensitive fields.
##########
utils/src/main/java/com/cloud/utils/nio/Link.java:
##########
@@ -354,16 +376,28 @@ public InetSocketAddress getSocketAddress() {
return _addr;
}
+ public Integer getLocalPort() {
+ return _localPort;
+ }
+
public String getIpAddress() {
return _addr.getAddress().toString();
}
public synchronized void terminated() {
+ if (LOGGER.isTraceEnabled()) {
+ LOGGER.debug("Terminating connection to {}", _addr);
+ }
_key = null;
Review Comment:
Logging-level mismatch: this is guarded by `isTraceEnabled()` but logs at DEBUG level. That makes the message appear at DEBUG even when TRACE is disabled (and also means the guard is ineffective).
##########
utils/src/main/java/com/cloud/utils/backoff/impl/ExponentialWithJitterBackoff.java:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.utils.backoff.impl;
+
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.NumbersUtil;
+import com.cloud.utils.backoff.BackoffAlgorithm;
+import com.cloud.utils.backoff.BackoffFactory;
+import com.cloud.utils.component.AdapterBase;
+
+import java.security.SecureRandom;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Exponential backoff with up/down cycling.
+ * Delay grows exponentially until a maximum, then decreases back to base, then repeats.
+ *
+ * @author mprokopchuk
+ */
+public class ExponentialWithJitterBackoff extends AdapterBase implements BackoffAlgorithm,
+ ExponentialWithJitterBackoffMBean {
+
+ /**
+ * Property name for the minimal delay to be used either by {@code agent.properties} file or by configuration key.
+ */
+ public static final String MIN_DELAY_MS_CONFIG_KEY = "backoff.min_delay_ms";
+
+ /**
+ * Property name for the maximal delay to be used either by {@code agent.properties} file or by configuration key.
+ */
+ public static final String MAX_DELAY_MS_CONFIG_KEY = "backoff.max_delay_ms";
+
+ /**
+ * Default value for minimal delay for the property {@link ExponentialWithJitterBackoff#MIN_DELAY_MS_DEFAULT}.
+ */
+ public static final int MIN_DELAY_MS_DEFAULT = 5_000;
+
+ /**
+ * Default value for maximal delay for the property {@link ExponentialWithJitterBackoff#MAX_DELAY_MS_DEFAULT}.
+ */
+ public static final int MAX_DELAY_MS_DEFAULT = 15_000;
+
+ private final Map<String, Thread> asleep = new ConcurrentHashMap<>();
+ private final Random random = new SecureRandom();
+
+ private int minDelayMs;
+ private int maxDelayMs;
+ private int maxAttempts;
+ private int attemptNumber;
+ private boolean increasing;
+
+ @Override
+ public void waitBeforeRetry() {
+ boolean interrupted = false;
+ long waitMs = getTimeToWait();
+ Thread current = Thread.currentThread();
+ try {
+ asleep.put(current.getName(), current);
+ logger.debug(String.format("Going to sleep for %s", DateUtil.formatMillis(waitMs)));
+ Thread.sleep(waitMs);
+ logger.debug(String.format("Sleep done for %s", DateUtil.formatMillis(waitMs)));
+ } catch (InterruptedException e) {
+ logger.info(String.format("Thread %s interrupted while waiting for retry", current.getName()), e);
+ } finally {
+ asleep.remove(current.getName());
+ calculateNextAttempt();
+ if (interrupted) {
+ Thread.currentThread().interrupt();
+ }
+ }
+ }
+
+ /**
+ * Calculates next attempt and direction.
+ */
+ private void calculateNextAttempt() {
+ if (increasing) {
+ int nextAttemptNumber = attemptNumber + 1;
+ increasing = getNextDelay() <= maxDelayMs && nextAttemptNumber <= maxAttempts;
+ if (increasing) {
+ attemptNumber = nextAttemptNumber;
+ }
+ } else {
+ int nextAttemptNumber = Math.max(attemptNumber - 1, 0);
+ increasing = nextAttemptNumber == 0;
+ if (!increasing) {
+ attemptNumber = nextAttemptNumber;
+ }
+ }
+ }
+
+ @Override
+ public void reset() {
+ attemptNumber = 0;
+ }
+
+ @Override
+ public Map<String, String> getConfiguration() {
+ Map<String, String> configuration = new HashMap<>();
+ configuration.put(MIN_DELAY_MS_CONFIG_KEY, String.valueOf(minDelayMs));
+ configuration.put(MAX_DELAY_MS_CONFIG_KEY, String.valueOf(maxDelayMs));
+ configuration.put(BackoffFactory.BACKOFF_IMPLEMENTATION_KEY, this.getClass().getName());
+ return configuration;
+ }
+
+ @Override
+ public boolean configure(String name, Map<String, Object> params) {
+ minDelayMs = NumbersUtil.parseInt((String) params.get(MIN_DELAY_MS_CONFIG_KEY), MIN_DELAY_MS_DEFAULT);
+ maxDelayMs = NumbersUtil.parseInt((String) params.get(MAX_DELAY_MS_CONFIG_KEY), MAX_DELAY_MS_DEFAULT);
+ maxAttempts = (int) Math.round(Math.log((double) maxDelayMs / minDelayMs) / Math.log(2));
+
+ attemptNumber = random.nextInt(maxAttempts + 1);
+ increasing = random.nextBoolean();
+ // do nothing
+ return true;
+ }
+
+ @Override
+ public Collection<String> getWaiters() {
+ return asleep.keySet();
+ }
+
+ @Override
+ public boolean wakeup(String threadName) {
+ Thread th = asleep.get(threadName);
+ if (th != null) {
+ th.interrupt();
+ return true;
+ }
+ return false;
+ }
+
+ private long getNextDelay() {
+ return (long) Math.min(minDelayMs * Math.pow(2, attemptNumber), maxDelayMs);
+ }
+
+ @Override
+ public long getTimeToWait() {
+ long delay = getNextDelay();
+ int jitter = random.nextInt((int) delay / 2);
+ return delay + jitter;
+ }
Review Comment:
`getTimeToWait()` can throw `IllegalArgumentException` when `delay / 2` is 0 (e.g., if `minDelayMs` is configured as 0 or 1). It also doesn't guard against invalid `minDelayMs/maxDelayMs` values (division by zero / negative delays).
##########
engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java:
##########
@@ -1376,62 +1763,129 @@ protected AgentAttache createAttacheForConnect(final HostVO host, final Link lin
return attache;
}
- private AgentAttache sendReadyAndGetAttache(HostVO host, ReadyCommand ready, Link link, StartupCommand[] startupCmds) throws ConnectionException {
+ private AgentAttache sendReadyAndGetAttache(HostVO host, ReadyCommand ready, Link link, StartupCommand[] startup) throws ConnectionException {
AgentAttache attache;
GlobalLock joinLock = getHostJoinLock(host.getId());
try {
- if (!joinLock.lock(60)) {
- throw new ConnectionException(true, String.format("Unable to acquire lock on host %s, to process agent connection", host));
+ long processStart = System.currentTimeMillis();
+ if (joinLock.lock(getTimeoutSec())) {
+ logProcessingStart(host, joinLock);
+ try {
+ updateReadyCommandWithMSList(host, ready, startup);
+ attache = createAndNotifyAttache(host, link, startup);
+ } finally {
+ logProcessingFinish(host, joinLock, processStart);
+ joinLock.unlock();
+ }
+ } else {
+ throw createLockAcquisitionException(host, joinLock, processStart);
}
-
- logger.debug("Acquired lock on host {}, to process agent connection", host);
- attache = connectHostAgent(host, ready, link, startupCmds, joinLock);
} finally {
joinLock.releaseRef();
}
return attache;
}
- private AgentAttache connectHostAgent(HostVO host, ReadyCommand ready, Link link, StartupCommand[] startupCmds, GlobalLock joinLock) throws ConnectionException {
- AgentAttache attache;
- try {
- final List<String> agentMSHostList = new ArrayList<>();
- String lbAlgorithm = null;
- if (startupCmds != null && startupCmds.length > 0) {
- final String agentMSHosts = startupCmds[0].getMsHostList();
- if (StringUtils.isNotEmpty(agentMSHosts)) {
- String[] msHosts = agentMSHosts.split("@");
- if (msHosts.length > 1) {
- lbAlgorithm = msHosts[1];
- }
- agentMSHostList.addAll(Arrays.asList(msHosts[0].split(",")));
+ private void logProcessingStart(HostVO host, GlobalLock joinLock) {
+ StringBuilder msgBuilder = getSummaryMsgBuilder("Processing Host connection started",
+ EventTypes.EVENT_HOST_RECONNECT, host.getUuid(), host.getName(), joinLock.getName(), null, null, null);
+ logger.info(msgBuilder.toString());
+ }
+
+ private void logProcessingFinish(HostVO host, GlobalLock joinLock, long processStart) {
+ long processFinish = System.currentTimeMillis() - processStart;
+ StringBuilder msgBuilder = getSummaryMsgBuilder("Processing Host connection finished",
+ EventTypes.EVENT_HOST_RECONNECT, host.getUuid(), host.getName(), joinLock.getName(), null, null,
+ processFinish);
+ logger.fatal(msgBuilder.toString());
Review Comment:
`logProcessingFinish` logs at FATAL level even on a normal successful connect path. This will likely pollute logs / monitoring and may trigger alerts despite success.
##########
engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java:
##########
@@ -1376,62 +1763,129 @@ protected AgentAttache createAttacheForConnect(final HostVO host, final Link lin
return attache;
}
- private AgentAttache sendReadyAndGetAttache(HostVO host, ReadyCommand ready, Link link, StartupCommand[] startupCmds) throws ConnectionException {
+ private AgentAttache sendReadyAndGetAttache(HostVO host, ReadyCommand ready, Link link, StartupCommand[] startup) throws ConnectionException {
AgentAttache attache;
GlobalLock joinLock = getHostJoinLock(host.getId());
try {
- if (!joinLock.lock(60)) {
- throw new ConnectionException(true, String.format("Unable to acquire lock on host %s, to process agent connection", host));
+ long processStart = System.currentTimeMillis();
+ if (joinLock.lock(getTimeoutSec())) {
+ logProcessingStart(host, joinLock);
+ try {
+ updateReadyCommandWithMSList(host, ready, startup);
+ attache = createAndNotifyAttache(host, link, startup);
+ } finally {
+ logProcessingFinish(host, joinLock, processStart);
+ joinLock.unlock();
+ }
+ } else {
+ throw createLockAcquisitionException(host, joinLock, processStart);
}
-
- logger.debug("Acquired lock on host {}, to process agent connection", host);
- attache = connectHostAgent(host, ready, link, startupCmds, joinLock);
} finally {
joinLock.releaseRef();
}
return attache;
}
- private AgentAttache connectHostAgent(HostVO host, ReadyCommand ready, Link link, StartupCommand[] startupCmds, GlobalLock joinLock) throws ConnectionException {
- AgentAttache attache;
- try {
- final List<String> agentMSHostList = new ArrayList<>();
- String lbAlgorithm = null;
- if (startupCmds != null && startupCmds.length > 0) {
- final String agentMSHosts = startupCmds[0].getMsHostList();
- if (StringUtils.isNotEmpty(agentMSHosts)) {
- String[] msHosts = agentMSHosts.split("@");
- if (msHosts.length > 1) {
- lbAlgorithm = msHosts[1];
- }
- agentMSHostList.addAll(Arrays.asList(msHosts[0].split(",")));
+ private void logProcessingStart(HostVO host, GlobalLock joinLock) {
+ StringBuilder msgBuilder = getSummaryMsgBuilder("Processing Host connection started",
+ EventTypes.EVENT_HOST_RECONNECT, host.getUuid(), host.getName(), joinLock.getName(), null, null, null);
+ logger.info(msgBuilder.toString());
+ }
+
+ private void logProcessingFinish(HostVO host, GlobalLock joinLock, long processStart) {
+ long processFinish = System.currentTimeMillis() - processStart;
+ StringBuilder msgBuilder = getSummaryMsgBuilder("Processing Host connection finished",
+ EventTypes.EVENT_HOST_RECONNECT, host.getUuid(), host.getName(), joinLock.getName(), null, null,
+ processFinish);
+ logger.fatal(msgBuilder.toString());
+ }
+
+ private void updateReadyCommandWithMSList(HostVO host, ReadyCommand ready, StartupCommand[] startup) {
+ List<String> agentMSHostList = new ArrayList<>();
+ String lbAlgorithm = null;
+
+ if (startup != null) {
+ String agentMSHosts = startup[0].getMsHostList();
+ if (StringUtils.isNotEmpty(agentMSHosts)) {
+ String[] msHosts = agentMSHosts.split("@");
+ if (msHosts.length > 1) {
+ lbAlgorithm = msHosts[1];
}
+ agentMSHostList.addAll(Arrays.asList(msHosts[0].split(",")));
}
+ }
- if (!indirectAgentLB.compareManagementServerListAndLBAlgorithm(host.getId(), host.getDataCenterId(), agentMSHostList, lbAlgorithm)) {
- final List<String> newMSList = indirectAgentLB.getManagementServerList(host.getId(), host.getDataCenterId(), null);
- ready.setMsHostList(newMSList);
- String newLBAlgorithm = indirectAgentLB.getLBAlgorithmName();
- ready.setLbAlgorithm(newLBAlgorithm);
- logger.debug("Agent's management server host list or lb algorithm is not up to date, sending list and algorithm update: {}, {}", newMSList, newLBAlgorithm);
- }
+ if (!indirectAgentLB.compareManagementServerListAndLBAlgorithm(host.getId(), host.getDataCenterId(), agentMSHostList, lbAlgorithm)) {
+ setReadyCommandMSList(host, ready);
+ }
+ }
- final List<String> avoidMsList = _mshostDao.listNonUpStateMsIPs();
- ready.setAvoidMsHostList(avoidMsList);
- ready.setLbCheckInterval(indirectAgentLB.getLBPreferredHostCheckInterval(host.getClusterId()));
- ready.setArch(host.getArch().getType());
+ private void setReadyCommandMSList(HostVO host, ReadyCommand ready) {
+ List<String> newMSList = indirectAgentLB.getManagementServerList(host.getId(), host.getDataCenterId(), null);
+ ready.setMsHostList(newMSList);
+ List<String> avoidMsList = _mshostDao.listNonUpStateMsIPs();
+ ready.setAvoidMsHostList(avoidMsList);
+ ready.setLbAlgorithm(indirectAgentLB.getLBAlgorithmName());
+ ready.setLbCheckInterval(indirectAgentLB.getLBPreferredHostCheckInterval(host.getClusterId()));
+ logger.debug("Agent's management server host list is not up to date, sending list update:" + newMSList);
+ }
- attache = createAttacheForConnect(host, link);
- attache = notifyMonitorsOfConnection(attache, startupCmds, false);
- } finally {
- joinLock.unlock();
+ private AgentAttache createAndNotifyAttache(HostVO host, Link link, StartupCommand[] startup) throws ConnectionException {
+ AgentAttache attache = createAttacheForConnect(host, link);
+ return notifyMonitorsOfConnection(attache, startup, false);
+ }
+
+ private ConnectionException createLockAcquisitionException(HostVO host, GlobalLock joinLock, long processStart) {
+ long processFinish = System.currentTimeMillis() - processStart;
+ String title = "Failed to connect Host";
+ String summary = String.format("Failure due to unable to acquire lock %s during %s seconds",
+ joinLock.getName(), 60);
+
Review Comment:
`createLockAcquisitionException` hard-codes `60` seconds in the summary, but the actual lock acquisition timeout is randomized via `getTimeoutSec()`. This makes the error summary misleading during troubleshooting.
##########
agent/src/main/java/com/cloud/agent/Agent.java:
##########
@@ -623,98 +559,351 @@ public Task create(final Task.Type type, final Link link, final byte[] data) {
return new ServerHandler(type, link, data);
}
- protected void reconnect(final Link link) {
- reconnect(link, null, false);
+ protected void closeAndTerminateLink(Link link) {
+ Optional.ofNullable(link)
+ .map(Link::attachment)
+ .filter(ServerAttache.class::isInstance)
+ .map(ServerAttache.class::cast)
+ .ifPresentOrElse(ServerAttache::disconnect, () -> {
+ if (link != null) {
+ link.close();
+ link.terminated();
+ }
+ });
}
- protected void reconnect(final Link link, String preferredMSHost, boolean forTransfer) {
- if (!(forTransfer || reconnectAllowed)) {
- logger.debug("Reconnect requested but it is not allowed {}", () -> getLinkLog(link));
+ protected void stopAndCleanupConnection() {
+ if (connection == null) {
return;
}
- cancelStartupTask();
- closeAndTerminateLink(link);
- closeAndTerminateLink(this.link);
- setLink(null);
- cancelTasks();
- serverResource.disconnected();
- logger.info("Lost connection to host: {}. Attempting reconnection while we still have {} commands in progress.", shell.getConnectedHost(), commandsInProgress.get());
- stopAndCleanupConnection(true);
- String host = preferredMSHost;
- if (org.apache.commons.lang3.StringUtils.isBlank(host)) {
- host = shell.getNextHost();
- }
- List<String> avoidMSHostList = shell.getAvoidHosts();
- do {
- if (CollectionUtils.isEmpty(avoidMSHostList) || !avoidMSHostList.contains(host)) {
- connection = new NioClient(getAgentName(), host, shell.getPort(), shell.getWorkers(), shell.getSslHandshakeTimeout(), this);
- logger.info("Reconnecting to host: {}", host);
- try {
- connection.start();
- } catch (final NioConnectionException e) {
- logger.info("Attempted to re-connect to the server, but received an unexpected exception, trying again...", e);
- stopAndCleanupConnection(false);
- }
+ NioConnection connection = this.connection;
+ connection.stop();
+ try {
+ connection.cleanUp();
+ } catch (final IOException e) {
+ logger.warn("Fail to clean up old connection", e);
+ }
+
+ try {
+ while (connection.isStartup()) {
+ logger.debug("Waiting for connection graceful stop");
+ shell.getBackoffAlgorithm().waitBeforeRetry();
+ connection.stop();
}
- shell.getBackoffAlgorithm().waitBeforeRetry();
- host = shell.getNextHost();
- } while (!connection.isStartup());
- shell.updateConnectedHost(((NioClient)connection).getHost());
- logger.info("Connected to the host: {}", shell.getConnectedHost());
+ } catch (Exception e) {
+ logger.warn("Failed to gracefully stop connection", e);
+ }
+ logger.debug("Connection stopped");
+ }
+
+ /**
+ * Select the host to reconnect to based on priority:
+ * 1. preferredHost if defined and not blank
+ * 2. Link's socket address IP if available and not null
+ * 3. shell.getNextHost() if the above two options are not met
+ *
+ * @param preferredHost the preferred host to connect to
+ * @param link the current link which may contain socket address information
+ * @return the host to connect to
+ */
+ protected String selectReconnectionHost(String preferredHost, Link link) {
+ return Optional.ofNullable(preferredHost)
+ .filter(org.apache.commons.lang3.StringUtils::isNotBlank)
+ .orElseGet(() -> Optional.ofNullable(link)
+ .map(Link::getSocketAddress)
+ .map(InetSocketAddress::getAddress)
+ .map(InetAddress::getHostAddress)
+ .orElseGet(shell::getNextHost));
}
- protected void closeAndTerminateLink(final Link link) {
- if (link == null) {
+ /**
+ * Reconnect to Management Server.
+ *
+ * @param link - connection holder
+ * @param preferredHost - if defined, reconnect will be performed to this Host first,
+ * otherwise will be used {@link IAgentShell#getNextHost()}
+ * @param forceReconnect - expected to be true if called by {@link MigrateAgentConnectionCommand},
+ * this is only "switch Management Server", it does not perform full Host Connect process.
+ */
+ protected void reconnect(Link link, String preferredHost, boolean forceReconnect) {
+ if (!reconnectLock.compareAndSet(false, true)) {
+ logger.warn("Reconnect is already running, exiting");
return;
}
- link.close();
- link.terminated();
+ String requestedLink = Optional.ofNullable(link).map(Link::toString).orElse("N/A");
+ String currentLink = Optional.ofNullable(this.link).map(Link::toString).orElse("N/A");
+ logger.info("Reconnect info: provided link: {}, agent link: {}, preferred host: {}, force" +
+ " reconnect: {}", requestedLink, currentLink, preferredHost, forceReconnect);
+
+ try {
+ logger.debug("Obtained reconnect lock");
+ if (!(forceReconnect || reconnectAllowed)) {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Reconnect requested but it is not allowed {}", link);
+ }
+ return;
+ }
+
+ if (isReconnectStormDetected(link, preferredHost, requestedLink, currentLink)) {
+ return;
+ }
+
+ cleanupConnectionBeforeReconnect(link);
+ // start with preferred host
+ String host = selectReconnectionHost(preferredHost, link);
+
+ String hostLog = LogUtils.getHostLog(host, shell.getPort());
+ List<String> avoidMsHostList = Optional.ofNullable(shell.getAvoidHosts()).orElseGet(List::of);
+ // pointer to the first element of "refuse loop"
+ AtomicReference<String> firstRefuseLoopHostRef = new AtomicReference<>(null);
+ // to break deadlock where "non-avoid" MS Hosts are down and only "avoid" are up
+ AtomicBoolean ignoreAvoidMsHostListRef = new AtomicBoolean(false);
+ do {
+ AtomicBoolean skipTimeoutRef = new AtomicBoolean(false);
+ String parentLogContextId = (String) ThreadContext.get("logcontextid");
+ if (parentLogContextId != null) {
+ ThreadContext.put("logcontextid-parent", parentLogContextId);
+ }
+ ThreadContext.put("logcontextid", UuidUtils.first(UUID.randomUUID().toString()));
+ if (ignoreAvoidMsHostListRef.get() || !avoidMsHostList.contains(host)) {
+ connection = new NioClient(getAgentName(), host, shell.getPort(), shell.getWorkers(),
+ shell.getSslHandshakeTimeout(), this);
+ logger.info("Reconnecting to host: {}", hostLog);
+ try {
+ connection.start();
+ // successfully connected, skip the rest
+ continue;
+ } catch (Exception e) {
+ logReconnectionFailure(e, hostLog);
+
+ try {
+ stopAndCleanupConnection();
+ } catch (Exception ex) {
+ logger.warn("Got an exception during stop and cleanup connection", e);
+ }
+
+ updateRefuseLoopState(e, host, firstRefuseLoopHostRef, ignoreAvoidMsHostListRef, skipTimeoutRef);
+ }
+ } else {
+ logger.debug("Next host {} is in avoid list, skipped", hostLog);
+ if (org.apache.commons.lang3.StringUtils.isBlank(preferredHost)) {
+ logHostLists(avoidMsHostList);
+ skipTimeoutRef.set(true);
+ }
+ }
+ if (!skipTimeoutRef.get()) {
+ shell.getBackoffAlgorithm().waitBeforeRetry();
+ }
+ host = shell.getNextHost();
+ hostLog = LogUtils.getHostLog(host, shell.getPort());
+ logger.debug("Next host to connect: {}", hostLog);
+ } while (!connection.isStartup());
+ // successfully connected
+ shell.updateConnectedHost(((NioClient) connection).getHost());
+ String msg = String.format("Connected to the host: %s (%s)", shell.getConnectedHost(), this.link);
+ logger.info(msg);
+ } finally {
+ reconnectLock.set(false);
+ logger.debug("Removed reconnect lock");
+ }
}
- protected void stopAndCleanupConnection(boolean waitForStop) {
- if (connection == null) {
- return;
+ /**
+ * Handles "Connection refused" loop detection and determines if backoff timeout should be skipped.
+ * Manages refuse loop state to detect when all management servers have been tried and need
+ * to ignore avoid list to prevent deadlock.
+ *
+ * @param e the exception from connection attempt
+ * @param host the current host being attempted
+ * @param firstRefuseLoopHostRef reference to first host in refuse loop (modified by this method)
+ * @param ignoreAvoidMsHostListRef flag to ignore avoid list (modified by this method)
+ * @return true if timeout should be skipped (connection refused), false otherwise
+ */
+ private void updateRefuseLoopState(Exception e, String host, AtomicReference<String> firstRefuseLoopHostRef, AtomicBoolean ignoreAvoidMsHostListRef, AtomicBoolean skipTimeoutRef) {
+ // we are skipping timeout for "Connection refused" to not waste time on down MS
+ boolean skipTimeout = Optional.ofNullable(e.getCause())
+ .filter(ConnectException.class::isInstance)
+ .map(Throwable::getMessage)
+ .filter(CONNECTION_REFUSED_MSG::equalsIgnoreCase)
+ .isPresent();
+ skipTimeoutRef.set(skipTimeout);
+ String firstRefuseLoopHost = firstRefuseLoopHostRef.get();
+ // for each "Connection refused" (maybe need to have a copy of variable with better name)
+ // start "refuse loop"
+ if (skipTimeout && firstRefuseLoopHost == null) {
+ firstRefuseLoopHostRef.set(host);
+ ignoreAvoidMsHostListRef.set(false);
+ logger.debug("Started refuse loop for host {}", firstRefuseLoopHost);
+ // closed "refuse loop"
+ } else if (skipTimeout && firstRefuseLoopHost.equalsIgnoreCase(host)) {
+ ignoreAvoidMsHostListRef.set(true);
+ logger.debug("Closed refuse loop for host {}", firstRefuseLoopHost);
+ // got non "refuse" related issue, break "refuse loop"
+ } else if (!skipTimeout && (firstRefuseLoopHostRef != null || ignoreAvoidMsHostListRef.get())) {
+ logger.debug("Broke refuse loop for host {} by {}", firstRefuseLoopHost, host);
+ firstRefuseLoopHostRef.set(null);
+ ignoreAvoidMsHostListRef.set(false);
}
- connection.stop();
+ }
+
+ /**
+ * Logs reconnection failure with appropriate level based on rejection reason.
+ * If connection was rejected due to max concurrent connections limit (Broken pipe),
+ * logs as warning. Otherwise logs as info.
+ *
+ * @param e the exception that occurred during reconnection attempt
+ * @param hostLog the formatted host log string for logging
+ */
+ private void logReconnectionFailure(Exception e, String hostLog) {
+ // check if got NIO Connection exception, caused by IO Exception "Broken pipe"
+ boolean rejectedByMs = Optional.of(e).filter(NioConnectionException.class::isInstance)
+ .map(Exception::getCause)
+ .filter(IOException.class::isInstance)
+ .map(IOException.class::cast)
+ .map(IOException::getMessage)
+ .filter(BROKEN_PIPE_MSG::equalsIgnoreCase)
+ .isPresent();
+ if (rejectedByMs) {
+ logger.warn("Attempted to re-connect to {}, but rejected" +
+ " due to 'agent.max.concurrent.new.connections' reached limit," +
+ " will try again", hostLog, e);
+ } else {
+ logger.info("Attempted to re-connect to {}, but got exception," +
+ " will try again", hostLog, e);
+ }
+ }
+
+ /**
+ * Logs all management server host lists for debugging reconnection logic.
+ * Outputs defined hosts, hosts to avoid, and calculated available hosts.
+ *
+ * @param avoidMsHostList list of management server hosts to avoid during reconnection
+ */
+ private void logHostLists(List<String> avoidMsHostList) {
+ logger.debug("Preferred host is not defined");
try {
- connection.cleanUp();
- } catch (final IOException e) {
- logger.warn("Fail to clean up old connection. {}", e);
+ List<String> hostsList = Optional.ofNullable(shell.getHosts())
+ .map(Arrays::asList)
+ .orElseGet(List::of);
+
+ List<String> hostsShortList = new ArrayList<>(hostsList);
+ hostsShortList.removeAll(avoidMsHostList);
+
+ logger.info("Defined hosts: {} Avoid hosts: {} Available hosts: {}",
+ String.join(", ", hostsList), String.join(", ", avoidMsHostList), String.join(", ", hostsShortList));
+ } catch (Exception e) {
+ logger.warn("Failed to calculate next host logic", e);
}
- if (!waitForStop) {
- return;
+ }
+
+ /**
+ * Cleans up current connection state before attempting reconnection.
+ * Stops host connect process, terminates links, cancels scheduled tasks,
+ * notifies server resource about disconnection, and resets connection tracking.
+ *
+ * @param link the link that triggered reconnection
+ */
+ private void cleanupConnectionBeforeReconnect(Link link) {
+ String lastConnectedHost = shell.getConnectedHost();
+ try {
+ // reset Host status track and Startup process initiating
+ logger.debug("Stopping Host Connect process");
+ hostConnectProcess.stop();
+ closeAndTerminateLink(link);
+ closeAndTerminateLink(this.link);
+ setLink(null);
+ cancelTasks();
+ serverResource.disconnected();
+ stopAndCleanupConnection();
+ shell.updateConnectedHost(null);
+ } catch (Exception ex) {
+ logger.error("Failed to cleanup previous connection", ex);
+ }
+ logger.info("Lost connection to host: {}. Attempting reconnection while we still have" +
+ " {} commands in progress.", lastConnectedHost, commandsInProgress.get());
+ }
+
+ /**
+ * Detects reconnection storm by checking if the reconnect request is redundant.
+ * This prevents processing stale reconnection requests for old links when
+ * agent has already established a new connection.
+ *
+ * @param link the link requesting reconnection
+ * @param preferredHost the preferred host to reconnect to (may be null)
+ * @param requestedLink string representation of the requested link for logging
+ * @param currentLink string representation of the current agent link for logging
+ * @return true if reconnection storm is detected and request should be skipped, false otherwise
+ */
+ private boolean isReconnectStormDetected(Link link, String preferredHost, String requestedLink, String currentLink) {
+ logger.debug("Calling storm guard");
+ boolean reconnectForCurrentLink = link == this.link;
+ boolean currentLinkTerminated = this.link != null && this.link.isTerminated();
+ boolean reconnectForNewHost = this.hostname != null && this.hostname.equals(preferredHost);
+ // if none of the above is true
+ boolean stormDetected = ! (reconnectForCurrentLink || currentLinkTerminated || reconnectForNewHost);
+ // connection storm guard
+ if (stormDetected) {
+ logger.warn("Reconnect requested for the connection {} but current connection is " +
+ "{} and preferred host {}, skipping", requestedLink, !currentLinkTerminated ? currentLink : currentLink + " (terminated)", preferredHost);
}
- do {
- shell.getBackoffAlgorithm().waitBeforeRetry();
- } while (connection.isStartup());
+ return stormDetected;
}
- public void processStartupAnswer(final Answer answer, final Response response, final Link link) {
- boolean answerValid = cancelStartupTask();
- final StartupAnswer startup = (StartupAnswer)answer;
+ public void processStartupAnswer(final StartupAnswer startup, final Response response, final Link link) {
+ setBackoffAlgorithm(startup);
+
if (!startup.getResult()) {
- logger.error("Not allowed to connect to the server: {}", answer.getDetails());
+ logger.error("Not allowed to connect to the server: {}", startup.getDetails());
if (serverResource != null && !serverResource.isExitOnFailures()) {
logger.trace("{} does not allow exit on failure, reconnecting",
serverResource.getClass().getSimpleName());
- reconnect(link);
+ logger.info("Reconnecting for {}", link);
+ requestHandler.submit(() -> reconnect(link, null, false));
return;
}
+ logger.fatal("Got unsuccessful result {} from the answer {}, details: {}",
+ startup.getResult(), startup.getClass().getSimpleName(), startup.getDetails());
System.exit(1);
}
- if (!answerValid) {
+
+ boolean processWasRunning = hostConnectProcess.stop();
+ if (!processWasRunning) {
logger.warn("Threw away a startup answer because we're reconnecting.");
return;
}
+ handleStartupAnswer(startup, response, link);
+ }
+
+ private void handleStartupAnswer(StartupAnswer startup, Response response, Link link) {
logger.info("Process agent startup answer, agent [id: {}, uuid: {}, name: {}] connected to the server",
startup.getHostId(), startup.getHostUuid(), startup.getHostName());
setId(startup.getHostId());
- setUuid(startup.getHostUuid());
- setName(startup.getHostName());
+ // older builds do not send host uuid and names, do not set it, otherwise null pointer exception will be thrown
+ String hostUuid = startup.getHostUuid();
+ if (org.apache.commons.lang3.StringUtils.isNotEmpty(hostUuid)) {
+ setUuid(hostUuid);
+ }
+ String hostName = startup.getHostName();
+ if (org.apache.commons.lang3.StringUtils.isNotEmpty(hostName)) {
+ setName(hostUuid);
+ }
Review Comment:
Bug: when `hostName` is present, the agent name is set to `hostUuid` instead of `hostName`, so the in-memory agent name becomes wrong (and may even become null if UUID was not provided).
##########
utils/src/main/java/com/cloud/utils/nio/NioConnection.java:
##########
@@ -80,66 +81,72 @@ public abstract class NioConnection implements Callable<Boolean> {
protected ExecutorService _executor;
protected ExecutorService _sslHandshakeExecutor;
protected CAService caService;
- protected Set<SocketChannel> socketChannels = ConcurrentHashMap.newKeySet();
- protected Integer sslHandshakeTimeout = null;
+ protected Set<SocketChannel> socketChannels;
+ protected Integer sslHandshakeTimeout;
private final int factoryMaxNewConnectionsCount;
- protected boolean blockNewConnections;
+ protected volatile boolean blockNewConnections;
public NioConnection(final String name, final int port, final int workers, final HandlerFactory factory) {
+ socketChannels = ConcurrentHashMap.newKeySet();
_name = name;
- _isRunning = false;
- blockNewConnections = false;
- _selector = null;
_port = port;
_workers = workers;
_factory = factory;
this.factoryMaxNewConnectionsCount = factory.getMaxConcurrentNewConnectionsCount();
- initWorkersExecutor();
- initSSLHandshakeExecutor();
}
- public void setCAService(final CAService caService) {
+ public void setCAService(CAService caService) {
this.caService = caService;
}
public void start() throws NioConnectionException {
_todos = new ArrayList<>();
+ if (_executor == null || _executor.isShutdown() || _executor.isTerminated()) {
+ initWorkersExecutor();
+ }
+
+ if (_sslHandshakeExecutor == null || _sslHandshakeExecutor.isShutdown() || _sslHandshakeExecutor.isTerminated()) {
+ initSSLHandshakeExecutor();
+ }
+
try {
init();
} catch (final ConnectException e) {
- logger.warn("Unable to connect to remote: is there a server running on port {}?", _port, e);
- throw new NioConnectionException(e.getMessage(), e);
- } catch (final IOException e) {
- logger.error("Unable to initialize the threads.", e);
- throw new NioConnectionException(e.getMessage(), e);
+ String msg = String.format("Unable to connect to remote: is there a server running on port %s: %s", _port, e.getMessage());
+ throw new NioConnectionException(msg, e);
} catch (final Exception e) {
- logger.error("Unable to initialize the threads due to unknown exception.", e);
- throw new NioConnectionException(e.getMessage(), e);
+ String msg = String.format("Unable to initialize the threads: %s", e.getMessage());
+ throw new NioConnectionException(msg, e);
}
_isStartup = true;
- if (_executor.isShutdown()) {
- initWorkersExecutor();
- }
- if (_sslHandshakeExecutor.isShutdown()) {
- initSSLHandshakeExecutor();
- }
_threadExecutor = Executors.newSingleThreadExecutor(new NamedThreadFactory(this._name + "-NioConnectionHandler"));
_isRunning = true;
+ // in case start() called after stop()
blockNewConnections = false;
_futureTask = _threadExecutor.submit(this);
}
public void stop() {
- _executor.shutdown();
- _sslHandshakeExecutor.shutdown();
- _isRunning = false;
- blockNewConnections = true;
+ if (_executor != null) {
+ if (logger.isTraceEnabled()) {
+ logger.trace("Shutting down handler tasks");
+ }
+ _executor.shutdownNow();
+ }
+ if (_sslHandshakeExecutor != null) {
+ if (logger.isTraceEnabled()) {
+ logger.trace("Shutting down SSL Handshake executor");
+ }
+ _sslHandshakeExecutor.shutdownNow();
+ }
if (_threadExecutor != null) {
_futureTask.cancel(false);
- _threadExecutor.shutdown();
+ _threadExecutor.shutdownNow();
}
Review Comment:
`stop()` cancels `_futureTask` without a null-check. Since executor initialization/submit is now deferred to `start()`, a caller that calls `stop()` after a failed/partial `start()` can hit an NPE here.
##########
utils/src/main/java/com/cloud/utils/backoff/impl/ExponentialWithJitterBackoff.java:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.utils.backoff.impl;
+
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.NumbersUtil;
+import com.cloud.utils.backoff.BackoffAlgorithm;
+import com.cloud.utils.backoff.BackoffFactory;
+import com.cloud.utils.component.AdapterBase;
+
+import java.security.SecureRandom;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Exponential backoff with up/down cycling.
+ * Delay grows exponentially until a maximum, then decreases back to base, then repeats.
+ *
+ * @author mprokopchuk
+ */
+public class ExponentialWithJitterBackoff extends AdapterBase implements BackoffAlgorithm,
+ ExponentialWithJitterBackoffMBean {
+
+ /**
+ * Property name for the minimal delay to be used either by {@code agent.properties} file or by configuration key.
+ */
+ public static final String MIN_DELAY_MS_CONFIG_KEY = "backoff.min_delay_ms";
+
+ /**
+ * Property name for the maximal delay to be used either by {@code agent.properties} file or by configuration key.
+ */
+ public static final String MAX_DELAY_MS_CONFIG_KEY = "backoff.max_delay_ms";
+
+ /**
+ * Default value for minimal delay for the property {@link ExponentialWithJitterBackoff#MIN_DELAY_MS_DEFAULT}.
+ */
+ public static final int MIN_DELAY_MS_DEFAULT = 5_000;
+
+ /**
+ * Default value for maximal delay for the property {@link ExponentialWithJitterBackoff#MAX_DELAY_MS_DEFAULT}.
+ */
+ public static final int MAX_DELAY_MS_DEFAULT = 15_000;
+
+ private final Map<String, Thread> asleep = new ConcurrentHashMap<>();
+ private final Random random = new SecureRandom();
+
+ private int minDelayMs;
+ private int maxDelayMs;
+ private int maxAttempts;
+ private int attemptNumber;
+ private boolean increasing;
+
+ @Override
+ public void waitBeforeRetry() {
+ boolean interrupted = false;
+ long waitMs = getTimeToWait();
+ Thread current = Thread.currentThread();
+ try {
+ asleep.put(current.getName(), current);
+ logger.debug(String.format("Going to sleep for %s", DateUtil.formatMillis(waitMs)));
+ Thread.sleep(waitMs);
+ logger.debug(String.format("Sleep done for %s", DateUtil.formatMillis(waitMs)));
+ } catch (InterruptedException e) {
+ logger.info(String.format("Thread %s interrupted while waiting for retry", current.getName()), e);
+ } finally {
+ asleep.remove(current.getName());
+ calculateNextAttempt();
+ if (interrupted) {
+ Thread.currentThread().interrupt();
+ }
+ }
Review Comment:
`waitBeforeRetry()` never re-sets the interrupt flag: `interrupted` is initialized to false and never set true in the catch block, so thread interruption gets swallowed.
##########
framework/db/src/main/java/com/cloud/utils/db/GlobalLock.java:
##########
@@ -45,127 +43,248 @@
* </p>
*/
public class GlobalLock {
- protected Logger logger = LogManager.getLogger(getClass());
+ protected final static Logger logger = LogManager.getLogger(GlobalLock.class);
private String name;
- private int lockCount = 0;
- private Thread ownerThread = null;
-
- private int referenceCount = 0;
- private long holdingStartTick = 0;
-
- private static Map<String, GlobalLock> s_lockMap = new HashMap<String, GlobalLock>();
+ /**
+ * DB lock count.
+ * Increments on {@link GlobalLock#lock(int)} and decrements on {@link GlobalLock#unlock()}.
+ * Upon {@link GlobalLock#unlock()}, if {@link GlobalLock#lockCount} is less than 1, then lock removed from DB
+ */
+ private int lockCount;
+
+ /**
+ * Internal (in-memory) lock count.
+ * Increments on {@link GlobalLock#addRef()} and indirectly on {@link GlobalLock#getInternLock(String)} and
+ * decrements on {@link GlobalLock#releaseRef()}, {@link GlobalLock#unlock()} and on {@link GlobalLock#lock(int)}
+ * if DB lock is unsuccessful
+ */
+ private int referenceCount;
+
+ /**
+ * Thread that owns lock. If lock called from different thread, it will be waiting for the owner to unlock it
+ * within requested timeout. If owner thread call {@link GlobalLock#lock(int)} again, then
+ * {@link GlobalLock#lockCount} will be incremented.
+ * If {@link GlobalLock#unlock()} called by owner thread, or DB lock will be unsuccessful, then owner thread will be
+ * nullified.
+ */
+ private Thread ownerThread;
+
+ /**
+ * Variable to hold lock duration in milliseconds. Used for information only.
+ */
+ private long holdingStartTick;
+
+ /**
+ * Holds all created locks.
+ */
+ private static Map<String, GlobalLock> s_lockMap = new HashMap<>();
+
+ /**
+ * Create lock.
+ *
+ * @param name lock name
+ */
private GlobalLock(String name) {
this.name = name;
}
+ /**
+ * Increment reference count to lock.
+ *
+ * @return reference count
+ */
public int addRef() {
synchronized (this) {
referenceCount++;
return referenceCount;
}
}
+ /**
+ * Decrement reference count to lock.
+ *
+ * @return reference count
+ */
public int releaseRef() {
- int refCount;
-
boolean needToRemove = false;
synchronized (this) {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Releasing reference for internal lock {}, reference count: {}, lock count: {}",
+ name, referenceCount, lockCount);
+ }
referenceCount--;
- refCount = referenceCount;
-
- if (referenceCount < 0)
- logger.warn("Unmatched Global lock " + name + " reference usage detected, check your code!");
- if (referenceCount == 0)
+ if (referenceCount < 0) {
+ logger.warn("Unmatched internal lock {} reference usage detected (reference count: {}, " +
+ "lock count: {}), check your code!", name, referenceCount, lockCount);
+ } else if (referenceCount < 1) {
needToRemove = true;
+ }
}
- if (needToRemove)
+ if (needToRemove) {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Need to release internal lock {}", name);
+ }
releaseInternLock(name);
+ }
+ if (logger.isDebugEnabled()) {
+ logger.debug("Released reference for lock {}, reference count: {}", name, referenceCount);
+ }
+ return referenceCount;
+ }
- return refCount;
+ public static boolean isLockAvailable(String name) {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Checking lock present for {}", name);
+ }
+ boolean result = false;
+ try {
+ result = DbUtil.isFreeLock(name);
+ } finally {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Result of checking lock present for {}: {}", name, result);
+ }
+ }
+ return result;
}
+ /**
+ * Registers internal lock (in memory) object. Does not create any lock in DB yet.
+ *
+ * @param name lock name
+ * @return lock object
+ */
public static GlobalLock getInternLock(String name) {
synchronized (s_lockMap) {
+ GlobalLock lock;
if (s_lockMap.containsKey(name)) {
- GlobalLock lock = s_lockMap.get(name);
- lock.addRef();
- return lock;
+ lock = s_lockMap.get(name);
+ if (logger.isDebugEnabled()) {
+ logger.debug("Internal lock {} already exists with reference count {} and lock count {}",
+ name, lock.referenceCount, lock.lockCount);
+ }
} else {
- GlobalLock lock = new GlobalLock(name);
- lock.addRef();
+ lock = new GlobalLock(name);
+ if (logger.isDebugEnabled()) {
+ logger.debug("Internal lock {} does not exist, adding", name);
+ }
s_lockMap.put(name, lock);
- return lock;
}
+ lock.addRef();
+ if (logger.isDebugEnabled()) {
+ logger.debug("Added reference to internal lock {}, reference count {}, lock count {}",
+ name, lock.referenceCount, lock.lockCount);
+ }
+ return lock;
}
}
+ /**
+ * Unregister internal lock (in memory) object. Does not remove any lock from DB.
+ *
+ * @param name lock name
+ */
private void releaseInternLock(String name) {
synchronized (s_lockMap) {
GlobalLock lock = s_lockMap.get(name);
if (lock != null) {
- if (lock.referenceCount == 0)
+ if (lock.referenceCount == 0) {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Released internal lock {}", name);
+ }
s_lockMap.remove(name);
+ } else {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Not releasing internal lock {} as it has references count: {}, lock count: {}",
+ name, lock.referenceCount, lock.lockCount);
+ }
+ }
} else {
- logger.warn("Releasing " + name + ", but it is already released.");
+ logger.warn("Internal lock {} already released", name);
}
}
}
+ /**
+ * Acquire or join existing DB lock.
+ *
+ * @param timeoutSeconds time in seconds during which lock needs to be obtained (it is not the lock duration)
+ * @return true if lock successfully obtained
+ */
public boolean lock(int timeoutSeconds) {
int remainingMilliSeconds = timeoutSeconds * 1000;
Profiler profiler = new Profiler();
boolean interrupted = false;
try {
while (true) {
synchronized (this) {
- if (ownerThread != null && ownerThread == Thread.currentThread()) {
- logger.warn("Global lock re-entrance detected");
-
+ if (ownerThread == Thread.currentThread()) {
+ logger.warn("Global lock {} re-entrance detected, owner thread: {}, reference count: {}, " +
+ "lock count: {}", getThreadName(ownerThread), name, referenceCount, lockCount);
+ // if it is re-entrance, then we may have more lock counts than needed?
Review Comment:
Logging bug: this message's placeholders don't match the arguments. It says "Global lock {lockName} ... owner thread {threadName}", but the first argument passed is the thread name and the second is the lock name, so the log line is swapped/misleading.
##########
core/src/main/java/com/cloud/resource/ServerResource.java:
##########
@@ -84,7 +84,8 @@ default StartupCommand[] initialize(boolean isTransferredConnection) {
void setAgentControl(IAgentControl agentControl);
default boolean isExitOnFailures() {
- return true;
+ // true would cause unnecessary Agent service restart, don't want it by default
+ return false;
}
Review Comment:
This changes the default behavior for *all* `ServerResource` implementations that don't override `isExitOnFailures()`. Previously the agent would exit by default; now it will keep running/reconnecting by default, which is a significant behavioral change beyond indirect-agent connection handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-4619584334
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-4620394267
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-5729763872
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-6466637188
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-4912845246
moving to 4.24.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-4620895411
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18149
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on code in PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#discussion_r3373466837
##########
core/src/main/java/com/cloud/agent/api/AgentConnectStatusCommand.java:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent.api;
+
+/**
+ * Command to check status of {@link StartupCommand} from the Agent.
+ *
+ * @author mprokopchuk
+ */
Review Comment:
@bernardodemarco I think, it's ok to have it. we've author mentioned in few other classes as well. (check with '`@author`' in the code base).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on code in PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#discussion_r3373466837
##########
core/src/main/java/com/cloud/agent/api/AgentConnectStatusCommand.java:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent.api;
+
+/**
+ * Command to check status of {@link StartupCommand} from the Agent.
+ *
+ * @author mprokopchuk
+ */
Review Comment:
@bernardodemarco I think, it's ok to have it. we've author mentioned in other classes as well. (check with '_@author_' in the code base).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#discussion_r4037511071
##########
agent/src/main/java/com/cloud/agent/AgentShell.java:
##########
@@ -424,11 +434,15 @@ public void init(String[] args) throws ConfigurationException {
_properties.put(cmdLineProp.getKey(), cmdLineProp.getValue());
}
- LOGGER.info("Defaulting to the constant time backoff algorithm");
- _backoff = new ConstantTimeBackoff();
- Map<String, Object> map = new HashMap<>();
- map.put("seconds", _properties.getProperty("backoff.seconds"));
- _backoff.configure("ConstantTimeBackoff", map);
+ try {
+ LOGGER.info("Creating backoff delay algorithm implementation");
+ setBackoffAlgorithm(BackoffFactory.create(_properties));
+ LOGGER.info("Created {} delay algorithm implementation", getBackoffAlgorithm().getClass().getName());
+ } catch (RuntimeException e) {
+ String msg = String.format("Failed to create backoff with provided properties %s, failing back to default", _properties);
Review Comment:
The fallback warning interpolates the entire `Properties` object, which can include the keystore passphrase and other secrets. A backoff configuration error should not cause unrelated agent credentials to be written to a warning log.
##########
engine/components-api/src/main/java/com/cloud/agent/AgentManager.java:
##########
@@ -178,4 +179,6 @@ enum TapAgentsAction {
boolean transferDirectAgentsFromMS(String fromMsUuid, long fromMsId, long timeoutDurationInMs, boolean excludeHostsInMaintenance);
int getHostSshPort(HostVO host);
+
+ List<String> getAvoidMsList();
Review Comment:
This adds a new abstract method to `AgentManager`, but `engine/storage/integration-test/src/test/java/org/apache/cloudstack/storage/test/DirectAgentManagerSimpleImpl.java` implements that interface without a `getAvoidMsList()` override. Compiling that integration-test module will fail; update the implementation (or make the interface method default if that is the intended contract).
##########
utils/src/main/java/com/cloud/utils/nio/Link.java:
##########
@@ -354,16 +376,28 @@ public InetSocketAddress getSocketAddress() {
return _addr;
}
+ public Integer getLocalPort() {
+ return _localPort;
+ }
+
public String getIpAddress() {
return _addr.getAddress().toString();
}
public synchronized void terminated() {
+ if (LOGGER.isTraceEnabled()) {
+ LOGGER.trace("Terminating connection to {}", _addr);
+ }
_key = null;
}
+ public boolean isTerminated() {
+ return _key == null;
+ }
Review Comment:
`isTerminated()` reads `_key` without synchronization, while `setKey()` and `terminated()` write it under the instance monitor. The reconnect and NIO threads can therefore observe a stale non-null key and process a link that has already been terminated. Make this read synchronized or make `_key` safely visible.
##########
framework/cluster/src/main/java/com/cloud/cluster/ClusterServiceServletImpl.java:
##########
@@ -146,15 +159,14 @@ private String executePostMethod(final CloseableHttpClient client, final HttpPos
result = EntityUtils.toString(httpResponse.getEntity());
profiler.stop();
if (logger.isDebugEnabled()) {
- logger.debug("POST " + serviceUrl + " response :" + result + ", responding time: " + profiler.getDurationInMillis() + " ms");
+ logger.debug("POST {} request: {}, response :{}, responding time: {} ms", serviceUrl, request, result, profiler.getDurationInMillis());
Review Comment:
This debug message logs the raw URL-encoded PDU request and response. `gsonPackage` contains arbitrary serialized agent commands and can carry sensitive values, so enabling debug logging exposes them in management-server logs. Log only method/sequence/timing metadata or redact both payloads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-5729779651
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-4625239434
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-5911610274
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-5925408997
@blueorangutan test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "kiranchavala (via GitHub)" <gi...@apache.org>.
kiranchavala commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-6034037585
@DaanHoogland @nvazquez can we merge this pr or any futher review is needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "nvazquez (via GitHub)" <gi...@apache.org>.
nvazquez commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-6388106862
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "Damans227 (via GitHub)" <gi...@apache.org>.
Damans227 commented on code in PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#discussion_r4162281815
##########
engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java:
##########
@@ -1253,15 +1635,16 @@ public Answer easySend(final Long hostId, final Command cmd) {
}
return answer;
-
+ // FIXME: There are a lot of result != null checks in callers. Should this method trow exception?
Review Comment:
ok thats gone now, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-6468809991
@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "bernardodemarco (via GitHub)" <gi...@apache.org>.
bernardodemarco commented on code in PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#discussion_r3365636892
##########
core/src/main/java/com/cloud/agent/api/AgentConnectStatusCommand.java:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent.api;
+
+/**
+ * Command to check status of {@link StartupCommand} from the Agent.
+ *
+ * @author mprokopchuk
+ */
Review Comment:
These `@author` fields are not necessary, right?
```suggestion
/**
* Command to check status of {@link StartupCommand} from the Agent.
*
*/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#discussion_r3354649209
##########
framework/db/src/main/java/com/cloud/utils/db/GlobalLock.java:
##########
@@ -45,127 +43,248 @@
* </p>
*/
public class GlobalLock {
- protected Logger logger = LogManager.getLogger(getClass());
+ protected final static Logger logger = LogManager.getLogger(GlobalLock.class);
private String name;
- private int lockCount = 0;
- private Thread ownerThread = null;
-
- private int referenceCount = 0;
- private long holdingStartTick = 0;
-
- private static Map<String, GlobalLock> s_lockMap = new HashMap<String, GlobalLock>();
+ /**
+ * DB lock count.
+ * Increments on {@link GlobalLock#lock(int)} and decrements on {@link GlobalLock#unlock()}.
+ * Upon {@link GlobalLock#unlock()}, if {@link GlobalLock#lockCount} is less than 1, then lock removed from DB
+ */
+ private int lockCount;
+
+ /**
+ * Internal (in-memory) lock count.
+ * Increments on {@link GlobalLock#addRef()} and indirectly on {@link GlobalLock#getInternLock(String)} and
+ * decrements on {@link GlobalLock#releaseRef()}, {@link GlobalLock#unlock()} and on {@link GlobalLock#lock(int)}
+ * if DB lock is unsuccessful
+ */
+ private int referenceCount;
+
+ /**
+ * Thread that owns lock. If lock called from different thread, it will be waiting for the owner to unlock it
+ * within requested timeout. If owner thread call {@link GlobalLock#lock(int)} again, then
+ * {@link GlobalLock#lockCount} will be incremented.
+ * If {@link GlobalLock#unlock()} called by owner thread, or DB lock will be unsuccessful, then owner thread will be
+ * nullified.
+ */
+ private Thread ownerThread;
+
+ /**
+ * Variable to hold lock duration in milliseconds. Used for information only.
+ */
+ private long holdingStartTick;
+
+ /**
+ * Holds all created locks.
+ */
+ private static Map<String, GlobalLock> s_lockMap = new HashMap<>();
+
+ /**
+ * Create lock.
+ *
+ * @param name lock name
+ */
private GlobalLock(String name) {
this.name = name;
}
+ /**
+ * Increment reference count to lock.
+ *
+ * @return reference count
+ */
public int addRef() {
synchronized (this) {
referenceCount++;
return referenceCount;
}
}
+ /**
+ * Decrement reference count to lock.
+ *
+ * @return reference count
+ */
public int releaseRef() {
- int refCount;
-
boolean needToRemove = false;
synchronized (this) {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Releasing reference for internal lock {}, reference count: {}, lock count: {}",
+ name, referenceCount, lockCount);
+ }
referenceCount--;
- refCount = referenceCount;
-
- if (referenceCount < 0)
- logger.warn("Unmatched Global lock " + name + " reference usage detected, check your code!");
- if (referenceCount == 0)
+ if (referenceCount < 0) {
+ logger.warn("Unmatched internal lock {} reference usage detected (reference count: {}, " +
+ "lock count: {}), check your code!", name, referenceCount, lockCount);
+ } else if (referenceCount < 1) {
needToRemove = true;
+ }
}
- if (needToRemove)
+ if (needToRemove) {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Need to release internal lock {}", name);
+ }
releaseInternLock(name);
+ }
+ if (logger.isDebugEnabled()) {
+ logger.debug("Released reference for lock {}, reference count: {}", name, referenceCount);
+ }
+ return referenceCount;
+ }
- return refCount;
+ public static boolean isLockAvailable(String name) {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Checking lock present for {}", name);
+ }
+ boolean result = false;
+ try {
+ result = DbUtil.isFreeLock(name);
+ } finally {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Result of checking lock present for {}: {}", name, result);
+ }
+ }
+ return result;
Review Comment:
GlobalLock.isLockAvailable() delegates to DbUtil.isFreeLock(), but the debug messages say "lock present" which is the opposite of what IS_FREE_LOCK() reports. This makes troubleshooting lock/availability logic confusing.
##########
engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java:
##########
@@ -1544,24 +2009,26 @@ protected void runInContext() {
}
protected void connectAgent(final Link link, final Command[] cmds, final Request request) {
- // send startupanswer to agent in the very beginning, so agent can move on without waiting for the answer for an undetermined time, if we put this logic into another
- // thread pool.
- final StartupAnswer[] answers = new StartupAnswer[cmds.length];
+ // send startup answer to agent in the very beginning, so agent can move on without waiting for the answer for an undetermined time,
+ // if we put this logic into another thread pool.
+ Map<String, String> backoffConfiguration = ConfigKeyUtil.toMap(BackoffConfiguration.value());
+ StartupAnswer[] answers = new StartupAnswer[cmds.length];
Command cmd;
for (int i = 0; i < cmds.length; i++) {
cmd = cmds[i];
- if (cmd instanceof StartupRoutingCommand || cmd instanceof StartupProxyCommand || cmd instanceof StartupSecondaryStorageCommand ||
- cmd instanceof StartupStorageCommand) {
- answers[i] = new StartupAnswer((StartupCommand) cmds[i], 0, "", "", mgmtServiceConf.getPingInterval());
- break;
+ if (cmd instanceof StartupRoutingCommand || cmd instanceof StartupProxyCommand || cmd instanceof StartupSecondaryStorageCommand
+ || cmd instanceof StartupStorageCommand) {
+ StartupAnswer answer = new StartupAnswer((StartupCommand) cmds[i], 0, "", "", mgmtServiceConf.getPingInterval());
+ answer.setParams(backoffConfiguration);
+ answer.setAgentHostStatusCheckDelaySec(AgentHostStatusCheckDelay.value());
+ answers[i] = answer;
}
}
- Response response;
- response = new Response(request, answers[0], _nodeId, -1);
+ Response response = new Response(request, answers, _nodeId, -1);
try {
Review Comment:
connectAgent() builds a StartupAnswer[] aligned to the incoming command indexes, but Response.getAnswer() always returns answers[0]. If the Startup*Command is not at index 0 (or if answers[0] remains null), the agent will not treat the response as a StartupAnswer and the connect handshake can fail silently.
##########
engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java:
##########
@@ -454,6 +580,45 @@ private AgentControlAnswer handleControlCommand(final AgentAttache attache, fina
return new AgentControlAnswer(cmd);
}
+ private AgentConnectStatusAnswer handleAgentConnectStatusCommand(AgentAttache attache, AgentConnectStatusCommand cmd) {
+ HostVO hostVo = _hostDao.findById(attache.getId());
+ return getConnectStatusAnswer(hostVo, cmd);
+ }
+
+ private AgentConnectStatusAnswer getConnectStatusAnswer(HostVO hostVo, AgentConnectStatusCommand cmd) {
+ long hostId = hostVo.getId();
+ String hostName = hostVo.getName();
+ String lockName = getHostJoinLockName(hostId);
+ Status status = hostVo.getStatus();
+ try {
+ boolean lockAvailable = GlobalLock.isLockAvailable(lockName);
+ String details = String.format("Global lock %s is%s present for %s", lockName, lockAvailable ? "" : " not",
+ hostName);
+ logger.debug(details);
+ return getAgentConnectStatusAnswer(cmd, lockName, hostName, lockAvailable, status, details);
+ } catch (RuntimeException e) {
+ String msg = String.format("Failed to check global lock %s presence for %s: %s", lockName, hostName,
+ e.getMessage());
+ logger.warn(msg, e);
+ return new AgentConnectStatusAnswer(cmd, false, msg);
+ }
Review Comment:
getConnectStatusAnswer() dereferences hostVo without a null-check (hostVo.getId()/getName()), but hostVo is sourced from HostDao lookups that can return null. This would turn an AgentConnectStatusCommand into an NPE and prevent the agent from determining whether it can proceed with StartupCommand.
##########
agent/src/test/java/com/cloud/agent/HostConnectProcessTest.java:
##########
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent;
+
+import com.cloud.exception.CloudException;
+import com.cloud.utils.nio.Link;
+import org.apache.logging.log4j.Logger;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.junit.MockitoJUnitRunner;
+import org.springframework.test.util.ReflectionTestUtils;
+
+import javax.naming.ConfigurationException;
+
+import static org.mockito.Mockito.mock;
+
+@RunWith(MockitoJUnitRunner.class)
+public class HostConnectProcessTest {
+
+ private Agent agent;
+ private Logger logger;
+ private Link link;
+ private ServerAttache attache;
+ private HostConnectProcess hostConnectProcess;
+ private boolean connectionTransfer;
+
+ @Before
+ public void setUp() throws ConfigurationException {
+ agent = mock(Agent.class);
+ logger = mock(Logger.class);
+ link = mock(Link.class);
+ attache = mock(ServerAttache.class);
+ hostConnectProcess = new HostConnectProcess(agent);
+ ReflectionTestUtils.setField(agent, "logger", logger);
+ }
+
+ @Test
+ public void testScheduleConnectProcess() throws InterruptedException, CloudException {
+
+ hostConnectProcess.scheduleConnectProcess(link, connectionTransfer);
+ Assert.assertTrue(hostConnectProcess.isInProgress());
+ }
+}
Review Comment:
HostConnectProcessTest schedules a ScheduledExecutorService (NamedThreadFactory creates non-daemon threads) but never shuts it down. This can leak threads across the test suite and cause tests to hang.
##########
agent/src/main/java/com/cloud/agent/HostConnectProcess.java:
##########
@@ -0,0 +1,355 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent;
+
+import com.cloud.agent.api.AgentConnectStatusAnswer;
+import com.cloud.agent.api.AgentConnectStatusCommand;
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.Command;
+import com.cloud.agent.api.StartupAnswer;
+import com.cloud.agent.api.StartupCommand;
+import com.cloud.agent.properties.AgentProperties;
+import com.cloud.agent.properties.AgentPropertiesFileHandler;
+import com.cloud.agent.transport.Request;
+import com.cloud.exception.CloudException;
+import com.cloud.exception.OperationTimedoutException;
+import com.cloud.host.Status;
+import com.cloud.resource.ResourceStatusUpdater;
+import com.cloud.resource.ServerResource;
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.cloud.utils.nio.Link;
+import org.apache.cloudstack.threadcontext.ThreadContextUtil;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.ThreadContext;
+
+import java.io.IOException;
+import java.nio.channels.ClosedChannelException;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Predicate;
+
+public class HostConnectProcess {
+ private static final Logger logger = LogManager.getLogger(HostConnectProcess.class);
+
+ public static final int DEFAULT_ASYNC_COMMAND_TIMEOUT_SEC =
+ AgentPropertiesFileHandler.getPropertyValue(AgentProperties.ASYNC_COMMAND_TIMEOUT_SEC);
+
+ public static final int DEFAULT_ASYNC_STARTUP_COMMAND_TIMEOUT_SEC =
+ AgentPropertiesFileHandler.getPropertyValue(AgentProperties.ASYNC_STARTUP_COMMAND_TIMEOUT_SEC);
+
+ static final long HOST_STATUS_CHECK_INITIAL_DELAY_SEC = 10;
+ private long hostStatusCheckDelaySec = AgentPropertiesFileHandler.getPropertyValue(AgentProperties.AGENT_HOST_STATUS_CHECK_DELAY_SEC);
+ private final AtomicReference<ScheduledFuture<?>> hostStatusFutureRef = new AtomicReference<>();
+ private final Agent agent;
+ private ScheduledExecutorService hostStatusExecutor;
+
+ public HostConnectProcess(Agent agent) {
+ this.agent = agent;
+ initExecutors();
+ }
+
+ private void initExecutors() {
+ stop();
+ var threadFactory = new NamedThreadFactory("Agent-" + HostStatusTask.class.getSimpleName());
+ hostStatusExecutor = Executors.newScheduledThreadPool(1, threadFactory);
+ }
+
+ /**
+ * Stops the whole connect process and cancels all scheduled asynchronous tasks.
+ * Returns {@link Boolean#TRUE} if {@link HostConnectProcess} was waiting for {@link StartupAnswer}.
+ */
+ public boolean stop() {
+ logger.debug("Stopping connect process. The process is active: {}", isInProgress());
+ stopHostStatusExecutor();
+ logger.debug("Stopped executor");
+ Optional<? extends ScheduledFuture<?>> hostStatusOpt = Optional.ofNullable(hostStatusFutureRef.getAndSet(null))
+ .filter(Predicate.not(ScheduledFuture::isCancelled));
+
+ hostStatusOpt.ifPresent(future -> future.cancel(true));
+ logger.debug("Cancelled future");
+
+ return hostStatusOpt.isPresent();
+ }
+
+ private void stopHostStatusExecutor() {
+ if (hostStatusExecutor != null) {
+ hostStatusExecutor.shutdownNow();
+ hostStatusExecutor = null;
+ }
+ }
+
+ public void scheduleConnectProcess(Link link, boolean connectionTransfer) {
+ logger.debug("Scheduling connect process for {}", link);
+ initExecutors();
+
+ var task = new HostStatusTask(link, connectionTransfer, agent, hostStatusFutureRef);
+ var future = hostStatusExecutor.scheduleWithFixedDelay(ThreadContextUtil.wrapThreadContext(task),
+ HOST_STATUS_CHECK_INITIAL_DELAY_SEC,
+ hostStatusCheckDelaySec, TimeUnit.SECONDS);
+ hostStatusFutureRef.set(future);
+ }
+
+ /**
+ * Returns {@link Boolean#TRUE} if {@link HostStatusTask} created and scheduled.
+ * That means there is already {@link Status#Connecting} process is running.
+ */
+ public boolean isInProgress() {
+ return Optional.ofNullable(hostStatusFutureRef.get())
+ .filter(Predicate.not(ScheduledFuture::isCancelled)).isPresent();
+ }
+
+ public void updateHostStatusCheckDelay(int newDelaySec) {
+ logger.info("Updating host status check delay from {} to {} seconds", hostStatusCheckDelaySec, newDelaySec);
+ this.hostStatusCheckDelaySec = newDelaySec;
+ }
+
+ /**
+ * Task wait for the Host to be available to connect to submit {@link StartupCommand}.
+ * Checks Host status on Management Server cluster and submit {@link StartupCommand} only if there is no lock and
+ * Host is not {@link Status#Connecting}.
+ */
+ public static class HostStatusTask implements Runnable, AsyncSend {
+ private final Set<Status> operationalStatuses = Set.of(Status.Connecting, Status.Up, Status.Rebalancing);
+
+ private final Link _link;
+ private final boolean _forceConnect;
+ private final Agent _agent;
+ private final AtomicReference<? extends ScheduledFuture<?>> _futureRef;
+
+ public HostStatusTask(Link link, boolean forceConnect, Agent agent,
+ AtomicReference<? extends ScheduledFuture<?>> futureRef) {
+ logger.debug("{} created", this.getClass().getSimpleName());
+ _link = link;
+ _forceConnect = forceConnect;
+ _agent = agent;
+ _futureRef = futureRef;
+ }
+
+ private void cancel() {
+ logger.debug("Cancelling future");
+ Optional.ofNullable(_futureRef.get())
+ .filter(Predicate.not(ScheduledFuture::isCancelled))
+ .ifPresent(future -> future.cancel(true));
+ logger.debug("Cancelled future");
+ }
+
+ @Override
+ public void run() {
+ try {
+ logger.debug("Running {}", getClass().getSimpleName());
+ runInternal();
+ } catch (Exception e) {
+ logger.error("Failed to run {}", getClass().getSimpleName(), e);
+ }
+ }
+
+ private void runInternal() {
+ ServerAttache attache = (ServerAttache) _link.attachment();
+ if (attache == null || attache.getLink() == null) {
+ cancel();
+ return;
+ }
+
+ AgentConnectStatusAnswer answer;
+ try {
+ answer = getAgentConnectStatusAnswer(attache);
+ } catch (IOException e) {
+ cancel();
+ logger.error("The connection to {} interrupted, restarting the whole process", _link, e);
+ _agent.getRequestHandler().submit(() -> _agent.reconnect(_link, null, _forceConnect));
+ return;
+ }
+ if (answer == null) {
+ logger.warn("Received empty agent connect status answer, will retry later");
+ return;
+ }
+ Boolean lockAvailable = answer.isLockAvailable();
+ Status status = answer.getHostStatus();
+ if (Boolean.TRUE.equals(lockAvailable)) {
+ // send startup command here
+ logger.info("There is no lock and Host status is {}", status);
+ try {
+ sendStartupCommand(_link, _forceConnect);
+ logger.debug("Sending startup command to {} finished", _link);
+ cancel();
+ logger.debug("Unscheduled {}", getClass().getSimpleName());
+ } catch (RuntimeException e) {
+ logger.error("Failed to send startup command to {}", _link, e);
+ } catch (IOException e) {
+ cancel();
+ logger.error("The connection to {} interrupted, restarting the whole process", _link, e);
+ _agent.getRequestHandler().submit(() -> _agent.reconnect(_link, null, _forceConnect));
+ }
+ } else {
+ logger.info("There is lock and Host status is {}, will retry later", status);
+ }
+ }
+
+ private AgentConnectStatusAnswer getAgentConnectStatusAnswer(ServerAttache attache) throws IOException {
+ AgentConnectStatusCommand command = _agent.setupAgentConnectStatusCommand(new AgentConnectStatusCommand());
+ var commands = new Command[]{command};
+ try {
+ return send(attache, commands, AgentConnectStatusAnswer.class, DEFAULT_ASYNC_COMMAND_TIMEOUT_SEC);
+ } catch (RuntimeException e) {
+ String commandName = commands[0].getClass().getSimpleName();
+ logger.error("Failed to retrieve {}, will retry later", commandName, e);
+ return null;
+ }
+ }
+
+ public void sendStartupCommand(Link link, boolean connectionTransfer) throws IOException {
+ ServerAttache attache = (ServerAttache) link.attachment();
+ if (attache == null || attache.getLink() == null) {
+ return;
+ }
+ ServerResource serverResource = _agent.getResource();
+ StartupCommand[] startup = serverResource.initialize();
+ if (ArrayUtils.isEmpty(startup)) {
Review Comment:
HostStatusTask.sendStartupCommand() ignores the connectionTransfer flag when building StartupCommand[] (it calls serverResource.initialize() instead of initialize(connectionTransfer)). Several ServerResource implementations override initialize(boolean) and may rely on this flag during initialization, not just for setting StartupCommand.connectionTransferred.
##########
agent/src/main/java/com/cloud/agent/Agent.java:
##########
@@ -623,98 +559,351 @@ public Task create(final Task.Type type, final Link link, final byte[] data) {
return new ServerHandler(type, link, data);
}
- protected void reconnect(final Link link) {
- reconnect(link, null, false);
+ protected void closeAndTerminateLink(Link link) {
+ Optional.ofNullable(link)
+ .map(Link::attachment)
+ .filter(ServerAttache.class::isInstance)
+ .map(ServerAttache.class::cast)
+ .ifPresentOrElse(ServerAttache::disconnect, () -> {
+ if (link != null) {
+ link.close();
+ link.terminated();
+ }
+ });
}
- protected void reconnect(final Link link, String preferredMSHost, boolean forTransfer) {
- if (!(forTransfer || reconnectAllowed)) {
- logger.debug("Reconnect requested but it is not allowed {}", () -> getLinkLog(link));
+ protected void stopAndCleanupConnection() {
+ if (connection == null) {
return;
}
- cancelStartupTask();
- closeAndTerminateLink(link);
- closeAndTerminateLink(this.link);
- setLink(null);
- cancelTasks();
- serverResource.disconnected();
- logger.info("Lost connection to host: {}. Attempting reconnection while we still have {} commands in progress.", shell.getConnectedHost(), commandsInProgress.get());
- stopAndCleanupConnection(true);
- String host = preferredMSHost;
- if (org.apache.commons.lang3.StringUtils.isBlank(host)) {
- host = shell.getNextHost();
- }
- List<String> avoidMSHostList = shell.getAvoidHosts();
- do {
- if (CollectionUtils.isEmpty(avoidMSHostList) || !avoidMSHostList.contains(host)) {
- connection = new NioClient(getAgentName(), host, shell.getPort(), shell.getWorkers(), shell.getSslHandshakeTimeout(), this);
- logger.info("Reconnecting to host: {}", host);
- try {
- connection.start();
- } catch (final NioConnectionException e) {
- logger.info("Attempted to re-connect to the server, but received an unexpected exception, trying again...", e);
- stopAndCleanupConnection(false);
- }
+ NioConnection connection = this.connection;
+ connection.stop();
+ try {
+ connection.cleanUp();
+ } catch (final IOException e) {
+ logger.warn("Fail to clean up old connection", e);
+ }
+
+ try {
+ while (connection.isStartup()) {
+ logger.debug("Waiting for connection graceful stop");
+ shell.getBackoffAlgorithm().waitBeforeRetry();
+ connection.stop();
}
- shell.getBackoffAlgorithm().waitBeforeRetry();
- host = shell.getNextHost();
- } while (!connection.isStartup());
- shell.updateConnectedHost(((NioClient)connection).getHost());
- logger.info("Connected to the host: {}", shell.getConnectedHost());
+ } catch (Exception e) {
+ logger.warn("Failed to gracefully stop connection", e);
+ }
+ logger.debug("Connection stopped");
+ }
+
+ /**
+ * Select the host to reconnect to based on priority:
+ * 1. preferredHost if defined and not blank
+ * 2. Link's socket address IP if available and not null
+ * 3. shell.getNextHost() if the above two options are not met
+ *
+ * @param preferredHost the preferred host to connect to
+ * @param link the current link which may contain socket address information
+ * @return the host to connect to
+ */
+ protected String selectReconnectionHost(String preferredHost, Link link) {
+ return Optional.ofNullable(preferredHost)
+ .filter(org.apache.commons.lang3.StringUtils::isNotBlank)
+ .orElseGet(() -> Optional.ofNullable(link)
+ .map(Link::getSocketAddress)
+ .map(InetSocketAddress::getAddress)
+ .map(InetAddress::getHostAddress)
+ .orElseGet(shell::getNextHost));
}
- protected void closeAndTerminateLink(final Link link) {
- if (link == null) {
+ /**
+ * Reconnect to Management Server.
+ *
+ * @param link - connection holder
+ * @param preferredHost - if defined, reconnect will be performed to this Host first,
+ * otherwise will be used {@link IAgentShell#getNextHost()}
+ * @param forceReconnect - expected to be true if called by {@link MigrateAgentConnectionCommand},
+ * this is only "switch Management Server", it does not perform full Host Connect process.
+ */
+ protected void reconnect(Link link, String preferredHost, boolean forceReconnect) {
+ if (!reconnectLock.compareAndSet(false, true)) {
+ logger.warn("Reconnect is already running, exiting");
return;
}
- link.close();
- link.terminated();
+ String requestedLink = Optional.ofNullable(link).map(Link::toString).orElse("N/A");
+ String currentLink = Optional.ofNullable(this.link).map(Link::toString).orElse("N/A");
+ logger.info("Reconnect info: provided link: {}, agent link: {}, preferred host: {}, force" +
+ " reconnect: {}", requestedLink, currentLink, preferredHost, forceReconnect);
+
+ try {
+ logger.debug("Obtained reconnect lock");
+ if (!(forceReconnect || reconnectAllowed)) {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Reconnect requested but it is not allowed {}", link);
+ }
+ return;
+ }
+
+ if (isReconnectStormDetected(link, preferredHost, requestedLink, currentLink)) {
+ return;
+ }
+
+ cleanupConnectionBeforeReconnect(link);
+ // start with preferred host
+ String host = selectReconnectionHost(preferredHost, link);
+
+ String hostLog = LogUtils.getHostLog(host, shell.getPort());
+ List<String> avoidMsHostList = Optional.ofNullable(shell.getAvoidHosts()).orElseGet(List::of);
+ // pointer to the first element of "refuse loop"
+ AtomicReference<String> firstRefuseLoopHostRef = new AtomicReference<>(null);
+ // to break deadlock where "non-avoid" MS Hosts are down and only "avoid" are up
+ AtomicBoolean ignoreAvoidMsHostListRef = new AtomicBoolean(false);
+ do {
+ AtomicBoolean skipTimeoutRef = new AtomicBoolean(false);
+ String parentLogContextId = (String) ThreadContext.get("logcontextid");
+ if (parentLogContextId != null) {
+ ThreadContext.put("logcontextid-parent", parentLogContextId);
+ }
+ ThreadContext.put("logcontextid", UuidUtils.first(UUID.randomUUID().toString()));
+ if (ignoreAvoidMsHostListRef.get() || !avoidMsHostList.contains(host)) {
+ connection = new NioClient(getAgentName(), host, shell.getPort(), shell.getWorkers(),
+ shell.getSslHandshakeTimeout(), this);
+ logger.info("Reconnecting to host: {}", hostLog);
+ try {
+ connection.start();
+ // successfully connected, skip the rest
+ continue;
+ } catch (Exception e) {
+ logReconnectionFailure(e, hostLog);
+
+ try {
+ stopAndCleanupConnection();
+ } catch (Exception ex) {
+ logger.warn("Got an exception during stop and cleanup connection", e);
+ }
+
+ updateRefuseLoopState(e, host, firstRefuseLoopHostRef, ignoreAvoidMsHostListRef, skipTimeoutRef);
+ }
+ } else {
+ logger.debug("Next host {} is in avoid list, skipped", hostLog);
+ if (org.apache.commons.lang3.StringUtils.isBlank(preferredHost)) {
+ logHostLists(avoidMsHostList);
+ skipTimeoutRef.set(true);
+ }
+ }
+ if (!skipTimeoutRef.get()) {
+ shell.getBackoffAlgorithm().waitBeforeRetry();
+ }
+ host = shell.getNextHost();
+ hostLog = LogUtils.getHostLog(host, shell.getPort());
+ logger.debug("Next host to connect: {}", hostLog);
+ } while (!connection.isStartup());
+ // successfully connected
+ shell.updateConnectedHost(((NioClient) connection).getHost());
+ String msg = String.format("Connected to the host: %s (%s)", shell.getConnectedHost(), this.link);
+ logger.info(msg);
+ } finally {
+ reconnectLock.set(false);
+ logger.debug("Removed reconnect lock");
+ }
}
- protected void stopAndCleanupConnection(boolean waitForStop) {
- if (connection == null) {
- return;
+ /**
+ * Handles "Connection refused" loop detection and determines if backoff timeout should be skipped.
+ * Manages refuse loop state to detect when all management servers have been tried and need
+ * to ignore avoid list to prevent deadlock.
+ *
+ * @param e the exception from connection attempt
+ * @param host the current host being attempted
+ * @param firstRefuseLoopHostRef reference to first host in refuse loop (modified by this method)
+ * @param ignoreAvoidMsHostListRef flag to ignore avoid list (modified by this method)
+ * @return true if timeout should be skipped (connection refused), false otherwise
+ */
+ private void updateRefuseLoopState(Exception e, String host, AtomicReference<String> firstRefuseLoopHostRef, AtomicBoolean ignoreAvoidMsHostListRef, AtomicBoolean skipTimeoutRef) {
+ // we are skipping timeout for "Connection refused" to not waste time on down MS
+ boolean skipTimeout = Optional.ofNullable(e.getCause())
+ .filter(ConnectException.class::isInstance)
+ .map(Throwable::getMessage)
+ .filter(CONNECTION_REFUSED_MSG::equalsIgnoreCase)
+ .isPresent();
+ skipTimeoutRef.set(skipTimeout);
+ String firstRefuseLoopHost = firstRefuseLoopHostRef.get();
+ // for each "Connection refused" (maybe need to have a copy of variable with better name)
+ // start "refuse loop"
+ if (skipTimeout && firstRefuseLoopHost == null) {
+ firstRefuseLoopHostRef.set(host);
+ ignoreAvoidMsHostListRef.set(false);
+ logger.debug("Started refuse loop for host {}", firstRefuseLoopHost);
+ // closed "refuse loop"
+ } else if (skipTimeout && firstRefuseLoopHost.equalsIgnoreCase(host)) {
+ ignoreAvoidMsHostListRef.set(true);
+ logger.debug("Closed refuse loop for host {}", firstRefuseLoopHost);
+ // got non "refuse" related issue, break "refuse loop"
+ } else if (!skipTimeout && (firstRefuseLoopHostRef != null || ignoreAvoidMsHostListRef.get())) {
+ logger.debug("Broke refuse loop for host {} by {}", firstRefuseLoopHost, host);
+ firstRefuseLoopHostRef.set(null);
+ ignoreAvoidMsHostListRef.set(false);
}
- connection.stop();
+ }
+
+ /**
+ * Logs reconnection failure with appropriate level based on rejection reason.
+ * If connection was rejected due to max concurrent connections limit (Broken pipe),
+ * logs as warning. Otherwise logs as info.
+ *
+ * @param e the exception that occurred during reconnection attempt
+ * @param hostLog the formatted host log string for logging
+ */
+ private void logReconnectionFailure(Exception e, String hostLog) {
+ // check if got NIO Connection exception, caused by IO Exception "Broken pipe"
+ boolean rejectedByMs = Optional.of(e).filter(NioConnectionException.class::isInstance)
+ .map(Exception::getCause)
+ .filter(IOException.class::isInstance)
+ .map(IOException.class::cast)
+ .map(IOException::getMessage)
+ .filter(BROKEN_PIPE_MSG::equalsIgnoreCase)
+ .isPresent();
+ if (rejectedByMs) {
+ logger.warn("Attempted to re-connect to {}, but rejected" +
+ " due to 'agent.max.concurrent.new.connections' reached limit," +
+ " will try again", hostLog, e);
+ } else {
+ logger.info("Attempted to re-connect to {}, but got exception," +
+ " will try again", hostLog, e);
+ }
+ }
+
+ /**
+ * Logs all management server host lists for debugging reconnection logic.
+ * Outputs defined hosts, hosts to avoid, and calculated available hosts.
+ *
+ * @param avoidMsHostList list of management server hosts to avoid during reconnection
+ */
+ private void logHostLists(List<String> avoidMsHostList) {
+ logger.debug("Preferred host is not defined");
try {
- connection.cleanUp();
- } catch (final IOException e) {
- logger.warn("Fail to clean up old connection. {}", e);
+ List<String> hostsList = Optional.ofNullable(shell.getHosts())
+ .map(Arrays::asList)
+ .orElseGet(List::of);
+
+ List<String> hostsShortList = new ArrayList<>(hostsList);
+ hostsShortList.removeAll(avoidMsHostList);
+
+ logger.info("Defined hosts: {} Avoid hosts: {} Available hosts: {}",
+ String.join(", ", hostsList), String.join(", ", avoidMsHostList), String.join(", ", hostsShortList));
+ } catch (Exception e) {
+ logger.warn("Failed to calculate next host logic", e);
}
- if (!waitForStop) {
- return;
+ }
+
+ /**
+ * Cleans up current connection state before attempting reconnection.
+ * Stops host connect process, terminates links, cancels scheduled tasks,
+ * notifies server resource about disconnection, and resets connection tracking.
+ *
+ * @param link the link that triggered reconnection
+ */
+ private void cleanupConnectionBeforeReconnect(Link link) {
+ String lastConnectedHost = shell.getConnectedHost();
+ try {
+ // reset Host status track and Startup process initiating
+ logger.debug("Stopping Host Connect process");
+ hostConnectProcess.stop();
+ closeAndTerminateLink(link);
+ closeAndTerminateLink(this.link);
+ setLink(null);
+ cancelTasks();
+ serverResource.disconnected();
+ stopAndCleanupConnection();
+ shell.updateConnectedHost(null);
+ } catch (Exception ex) {
+ logger.error("Failed to cleanup previous connection", ex);
+ }
+ logger.info("Lost connection to host: {}. Attempting reconnection while we still have" +
+ " {} commands in progress.", lastConnectedHost, commandsInProgress.get());
+ }
+
+ /**
+ * Detects reconnection storm by checking if the reconnect request is redundant.
+ * This prevents processing stale reconnection requests for old links when
+ * agent has already established a new connection.
+ *
+ * @param link the link requesting reconnection
+ * @param preferredHost the preferred host to reconnect to (may be null)
+ * @param requestedLink string representation of the requested link for logging
+ * @param currentLink string representation of the current agent link for logging
+ * @return true if reconnection storm is detected and request should be skipped, false otherwise
+ */
+ private boolean isReconnectStormDetected(Link link, String preferredHost, String requestedLink, String currentLink) {
+ logger.debug("Calling storm guard");
+ boolean reconnectForCurrentLink = link == this.link;
+ boolean currentLinkTerminated = this.link != null && this.link.isTerminated();
+ boolean reconnectForNewHost = this.hostname != null && this.hostname.equals(preferredHost);
+ // if none of the above is true
+ boolean stormDetected = ! (reconnectForCurrentLink || currentLinkTerminated || reconnectForNewHost);
Review Comment:
isReconnectStormDetected() treats a reconnect as "not a storm" when preferredHost equals this.hostname (the agent's hostname). preferredHost is a Management Server host, so this condition is unrelated and can incorrectly cause legitimate reconnects to be skipped (or storms to be misdetected).
##########
utils/src/main/java/com/cloud/utils/LogUtils.java:
##########
@@ -106,4 +108,29 @@ public static String logGsonWithoutException(String formatMessage, Object ... ob
return errorMsg;
}
}
+
+ /**
+ * Generates address entry for log in format of {@code IP_ADDRESS/HOST_NAME:PORT}, where {@code HOST_NAME} is
+ * optional if it cannot be resolved.
+ *
+ * @param address IP address or Host name
+ * @param port port
+ */
+ public static String getHostLog(String address, Integer port) {
+ try {
+ InetAddress inetAddress = InetAddress.getByName(address);
+ String hostName = inetAddress.getHostName();
+ String ipAddress = inetAddress.getHostAddress();
+ if (port == null) {
+ return String.format("%s/%s", ipAddress, hostName);
+ }
+ return String.format("%s/%s:%s", ipAddress, hostName, port);
+ } catch (UnknownHostException e) {
+ LOGGER.warn("Failed to resolve name for address {}", address, e);
+ }
+ if (port == null) {
+ return address;
+ }
+ return String.format("%s:%s", address, port);
+ }
Review Comment:
LogUtils.getHostLog() performs DNS lookups (InetAddress.getByName + getHostName). This is called during connection/reconnection logging (e.g., NioClient.init and Agent.reconnect loop), so slow/blocked DNS can materially delay reconnect attempts. Consider avoiding reverse lookups and just log the resolved IP plus the original host string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-4646256003
This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-4649382230
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#discussion_r4368124661
##########
utils/src/main/java/com/cloud/utils/backoff/impl/ExponentialWithJitterBackoff.java:
##########
@@ -0,0 +1,168 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.utils.backoff.impl;
+
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.NumbersUtil;
+import com.cloud.utils.backoff.BackoffAlgorithm;
+import com.cloud.utils.backoff.BackoffFactory;
+import com.cloud.utils.component.AdapterBase;
+
+import java.security.SecureRandom;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Exponential backoff with up/down cycling.
+ * Delay grows exponentially until a maximum, then decreases back to base, then repeats.
+ *
+ * @author mprokopchuk
+ */
+public class ExponentialWithJitterBackoff extends AdapterBase implements BackoffAlgorithm,
+ ExponentialWithJitterBackoffMBean {
+
+ /**
+ * Property name for the minimal delay to be used either by {@code agent.properties} file or by configuration key.
+ */
+ public static final String MIN_DELAY_MS_CONFIG_KEY = "backoff.min_delay_ms";
+
+ /**
+ * Property name for the maximal delay to be used either by {@code agent.properties} file or by configuration key.
+ */
+ public static final String MAX_DELAY_MS_CONFIG_KEY = "backoff.max_delay_ms";
+
+ /**
+ * Default value for minimal delay for the property {@link ExponentialWithJitterBackoff#MIN_DELAY_MS_DEFAULT}.
+ */
+ public static final int MIN_DELAY_MS_DEFAULT = 5_000;
+
+ /**
+ * Default value for maximal delay for the property {@link ExponentialWithJitterBackoff#MAX_DELAY_MS_DEFAULT}.
+ */
+ public static final int MAX_DELAY_MS_DEFAULT = 15_000;
+
+ private final Map<String, Thread> asleep = new ConcurrentHashMap<>();
+ private final Random random = new SecureRandom();
+
+ private int minDelayMs;
+ private int maxDelayMs;
+ private int maxAttempts;
+ private int attemptNumber;
+ private boolean increasing;
+
+ @Override
+ public void waitBeforeRetry() {
+ boolean interrupted = false;
+ long waitMs = getTimeToWait();
+ Thread current = Thread.currentThread();
+ try {
+ asleep.put(current.getName(), current);
+ logger.debug("Going to sleep for {}", DateUtil.formatMillis(waitMs));
+ Thread.sleep(waitMs);
+ logger.debug("Sleep done for {}", DateUtil.formatMillis(waitMs));
+ } catch (InterruptedException e) {
+ interrupted = true;
+ logger.info("Thread {} interrupted while waiting for retry", current.getName(), e);
+ } finally {
+ asleep.remove(current.getName());
+ calculateNextAttempt();
Review Comment:
The backoff instance is shared by the agent's reconnect workers, but `attemptNumber` and `increasing` are read and updated without synchronization. Concurrent `waitBeforeRetry()` calls can lose updates or calculate delays from inconsistent state, defeating the exponential backoff that is intended to prevent connection storms. Make the state transition atomic or keep attempt state per connection/retry process.
##########
framework/cluster/src/main/java/com/cloud/cluster/ManagementServerAddressUtil.java:
##########
@@ -0,0 +1,216 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.cluster;
+
+import org.apache.cloudstack.config.ApiServiceConfiguration;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import com.cloud.utils.net.NetUtils;
+
+import java.net.InetAddress;
+import java.net.UnknownHostException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.TimeUnit;
+import java.util.regex.Pattern;
+
+/**
+ * Utility class for management server address configuration detection.
+ */
+public class ManagementServerAddressUtil {
+ private static final Logger logger = LogManager.getLogger(ManagementServerAddressUtil.class);
+
+ private static final long RESOLUTION_CACHE_TTL_MS = TimeUnit.MINUTES.toMillis(5);
+ private static final Map<String, ResolvedAddress> resolutionCache = new ConcurrentHashMap<>();
+
+ private static final class ResolvedAddress {
+ private final String ip;
+ private final long resolvedAt;
+
+ private ResolvedAddress(String ip, long resolvedAt) {
+ this.ip = ip;
+ this.resolvedAt = resolvedAt;
+ }
+ }
+
+ // RFC 1123 compliant hostname pattern
+ private static final Pattern HOSTNAME_PATTERN = Pattern.compile(
+ "^([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\\-]*[a-zA-Z0-9])(\\.([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\\-]*[a-zA-Z0-9]))*$"
+ );
+
+ /**
+ * Validates if a string is a valid hostname according to RFC 1123.
+ *
+ * @param hostname the string to validate
+ * @return true if the string is a valid hostname format, false otherwise
+ */
+ private static boolean isValidHostname(String hostname) {
+ if (StringUtils.isEmpty(hostname) || hostname.length() > 253) {
+ return false;
+ }
+
+ // Check each label doesn't exceed 63 characters
+ String[] labels = hostname.split("\\.");
+ for (String label : labels) {
+ if (label.length() > 63) {
+ return false;
+ }
+ }
+
+ return HOSTNAME_PATTERN.matcher(hostname).matches();
+ }
+
+ /**
+ * Detects if the management server address list configuration uses hostnames or IP addresses.
+ * Checks all entries in the 'host' configuration (ApiServiceConfiguration.ManagementServerAddresses)
+ * to determine the format.
+ *
+ * @return true if ALL addresses are valid hostnames, false otherwise (including if any are IPs or invalid)
+ */
+ public static boolean isManagementServerAddressListUsingHostnames() {
+ final String msServerAddresses = ApiServiceConfiguration.ManagementServerAddresses.value();
+ if (StringUtils.isEmpty(msServerAddresses)) {
+ return false; // Default to IP format for safety
+ }
+
+ final String[] addresses = msServerAddresses.replace(" ", "").split(",");
+
+ boolean hasHostname = false;
+ for (String address : addresses) {
+ if (StringUtils.isEmpty(address)) {
+ continue;
+ }
+
+ // If it's an IP address, return false
+ if (NetUtils.isValidIp4(address) || NetUtils.isValidIp6(address)) {
+ return false;
+ }
+
+ // If it's not a valid hostname format, return false
+ if (!isValidHostname(address)) {
+ return false;
+ }
+
+ hasHostname = true;
+ }
+
+ // Only treat as hostname format if at least one valid hostname was found
+ return hasHostname;
+ }
+
+ /**
+ * Returns the entries of the 'host' configuration (ApiServiceConfiguration.ManagementServerAddresses) as a list,
+ * in the configured order.
+ */
+ public static List<String> getConfiguredAddressList() {
+ final String msServerAddresses = ApiServiceConfiguration.ManagementServerAddresses.value();
+ if (StringUtils.isEmpty(msServerAddresses)) {
+ return Collections.emptyList();
+ }
+ final List<String> addresses = new ArrayList<>();
+ for (String address : msServerAddresses.replace(" ", "").split(",")) {
+ if (StringUtils.isNotEmpty(address)) {
+ addresses.add(address);
+ }
+ }
+ return addresses;
+ }
+
+ /**
+ * Convenience overload of {@link #getConfiguredAddresses(Collection, Collection)} for a single management server.
+ */
+ public static List<String> getConfiguredAddresses(String msHostname, String msIp) {
+ return getConfiguredAddresses(msHostname != null ? List.of(msHostname) : List.of(),
+ msIp != null ? List.of(msIp) : List.of());
+ }
+
+ /**
+ * Returns the entries of the 'host' configuration that refer to any of the given management servers.
+ * Agents only know the configured entries, so lists sent to them (avoid list, maintenance list) must use
+ * this namespace rather than the hostnames and IPs persisted in the mshost table.
+ * A configured entry matches when it equals one of the persisted IPs, equals one of the persisted hostnames
+ * (case-insensitive) or, for a hostname alias, resolves to one of the persisted IPs. Resolutions are cached
+ * for {@value #RESOLUTION_CACHE_TTL_MS} ms so this can be called on hot paths such as ping answers.
+ *
+ * @param msHostnames hostnames of the management servers as persisted in the mshost table
+ * @param msIps service IPs of the management servers as persisted in the mshost table
+ * @return the matching configured entries, in the configured order; never null
+ */
+ public static List<String> getConfiguredAddresses(Collection<String> msHostnames, Collection<String> msIps) {
+ final Set<String> names = new HashSet<>();
+ if (msHostnames != null) {
+ for (String name : msHostnames) {
+ if (StringUtils.isNotBlank(name)) {
+ names.add(name.toLowerCase());
+ }
+ }
+ }
+ final Set<String> ips = new HashSet<>();
+ if (msIps != null) {
+ for (String ip : msIps) {
+ if (StringUtils.isNotBlank(ip)) {
+ ips.add(ip);
+ }
+ }
+ }
+ final List<String> matches = new ArrayList<>();
+ if (names.isEmpty() && ips.isEmpty()) {
+ return matches;
+ }
+ for (String address : getConfiguredAddressList()) {
+ if (ips.contains(address) || names.contains(address.toLowerCase())) {
+ matches.add(address);
+ continue;
+ }
+ if (ips.isEmpty() || NetUtils.isValidIp4(address) || NetUtils.isValidIp6(address)) {
+ continue;
+ }
+ final String resolvedIp = resolve(address);
+ if (resolvedIp != null && ips.contains(resolvedIp)) {
+ matches.add(address);
+ }
+ }
+ return matches;
+ }
+
+ private static String resolve(String hostname) {
+ final long now = System.currentTimeMillis();
+ final ResolvedAddress cached = resolutionCache.get(hostname);
+ if (cached != null && now - cached.resolvedAt < RESOLUTION_CACHE_TTL_MS) {
+ return cached.ip;
+ }
+ String ip = null;
+ try {
+ ip = InetAddress.getByName(hostname).getHostAddress();
+ } catch (UnknownHostException e) {
+ logger.debug("Unable to resolve configured management server address {}", hostname);
+ }
+ resolutionCache.put(hostname, new ResolvedAddress(ip, now));
Review Comment:
When a configured alias cannot be resolved, `ip` remains null and this `ConcurrentHashMap.put` throws a NullPointerException (which is not caught). This makes `getConfiguredAddresses` fail instead of skipping the unresolvable alias, and the added `testConfiguredAddressesSkipUnresolvableAliases` exercises exactly that path. Only cache successful resolutions (or use a null-capable representation).
##########
utils/src/main/java/com/cloud/utils/nio/NioConnection.java:
##########
@@ -229,7 +240,18 @@ protected boolean rejectConnectionIfBlocked(final SocketChannel socketChannel) t
if (!blockNewConnections) {
return false;
}
- logger.warn("Rejecting new connection as the server is blocked from accepting new connections");
+
+ try {
+ socketChannel.shutdownOutput();
+ } catch (IOException ignore) {
+ // ignore exception
+ }
+ try {
+ socketChannel.shutdownInput();
+ } catch (IOException ignore) {
+ // ignore exception
+ }
+ logger.warn("Rejecting new connection from {} as the server is blocked from accepting new connections", socketChannel.getRemoteAddress());
Review Comment:
`SocketChannel.getRemoteAddress()` is allowed to throw `IOException`, and logger argument evaluation happens before `warn` is called. If the newly accepted channel is closed during this rejection path, that exception escapes `accept()` and the selector loop catches it as fatal, stopping the NIO server before the channel is closed. Use the socket's non-throwing remote socket address or catch the lookup failure so a rejected connection cannot terminate the server.
This issue also appears on line 275 of the same file.
##########
utils/src/main/java/com/cloud/utils/backoff/impl/ExponentialWithJitterBackoff.java:
##########
@@ -0,0 +1,168 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.utils.backoff.impl;
+
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.NumbersUtil;
+import com.cloud.utils.backoff.BackoffAlgorithm;
+import com.cloud.utils.backoff.BackoffFactory;
+import com.cloud.utils.component.AdapterBase;
+
+import java.security.SecureRandom;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Exponential backoff with up/down cycling.
+ * Delay grows exponentially until a maximum, then decreases back to base, then repeats.
+ *
+ * @author mprokopchuk
+ */
+public class ExponentialWithJitterBackoff extends AdapterBase implements BackoffAlgorithm,
+ ExponentialWithJitterBackoffMBean {
+
+ /**
+ * Property name for the minimal delay to be used either by {@code agent.properties} file or by configuration key.
+ */
+ public static final String MIN_DELAY_MS_CONFIG_KEY = "backoff.min_delay_ms";
+
+ /**
+ * Property name for the maximal delay to be used either by {@code agent.properties} file or by configuration key.
+ */
+ public static final String MAX_DELAY_MS_CONFIG_KEY = "backoff.max_delay_ms";
+
+ /**
+ * Default value for minimal delay for the property {@link ExponentialWithJitterBackoff#MIN_DELAY_MS_DEFAULT}.
+ */
+ public static final int MIN_DELAY_MS_DEFAULT = 5_000;
+
+ /**
+ * Default value for maximal delay for the property {@link ExponentialWithJitterBackoff#MAX_DELAY_MS_DEFAULT}.
+ */
+ public static final int MAX_DELAY_MS_DEFAULT = 15_000;
+
+ private final Map<String, Thread> asleep = new ConcurrentHashMap<>();
+ private final Random random = new SecureRandom();
+
+ private int minDelayMs;
+ private int maxDelayMs;
+ private int maxAttempts;
+ private int attemptNumber;
+ private boolean increasing;
+
+ @Override
+ public void waitBeforeRetry() {
+ boolean interrupted = false;
+ long waitMs = getTimeToWait();
+ Thread current = Thread.currentThread();
+ try {
+ asleep.put(current.getName(), current);
+ logger.debug("Going to sleep for {}", DateUtil.formatMillis(waitMs));
+ Thread.sleep(waitMs);
+ logger.debug("Sleep done for {}", DateUtil.formatMillis(waitMs));
+ } catch (InterruptedException e) {
+ interrupted = true;
+ logger.info("Thread {} interrupted while waiting for retry", current.getName(), e);
+ } finally {
+ asleep.remove(current.getName());
+ calculateNextAttempt();
+ if (interrupted) {
+ Thread.currentThread().interrupt();
+ }
+ }
+ }
+
+ /**
+ * Calculates next attempt and direction.
+ */
+ private void calculateNextAttempt() {
+ if (increasing) {
+ int nextAttemptNumber = attemptNumber + 1;
+ increasing = getNextDelay() <= maxDelayMs && nextAttemptNumber <= maxAttempts;
+ if (increasing) {
+ attemptNumber = nextAttemptNumber;
+ }
+ } else {
+ int nextAttemptNumber = Math.max(attemptNumber - 1, 0);
+ increasing = nextAttemptNumber == 0;
+ if (!increasing) {
+ attemptNumber = nextAttemptNumber;
+ }
Review Comment:
When the decreasing phase reaches zero, `attemptNumber` is not updated because the assignment is inside `if (!increasing)`. For any cycle that reaches an attempt above zero, the next cycle therefore starts at attempt 1 instead of the configured base delay, contradicting the documented return-to-base behavior and producing an incorrect backoff sequence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-6158397434
@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-6464535803
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-4619580382
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#discussion_r3864133621
##########
agent/src/main/java/com/cloud/agent/Agent.java:
##########
@@ -623,98 +559,363 @@ public Task create(final Task.Type type, final Link link, final byte[] data) {
return new ServerHandler(type, link, data);
}
- protected void reconnect(final Link link) {
- reconnect(link, null, false);
+ protected void closeAndTerminateLink(Link link) {
+ Optional.ofNullable(link)
+ .map(Link::attachment)
+ .filter(ServerAttache.class::isInstance)
+ .map(ServerAttache.class::cast)
+ .ifPresentOrElse(ServerAttache::disconnect, () -> {
+ if (link != null) {
+ link.close();
+ link.terminated();
+ }
+ });
}
- protected void reconnect(final Link link, String preferredMSHost, boolean forTransfer) {
- if (!(forTransfer || reconnectAllowed)) {
- logger.debug("Reconnect requested but it is not allowed {}", () -> getLinkLog(link));
+ protected void stopAndCleanupConnection() {
+ if (connection == null) {
return;
}
- cancelStartupTask();
- closeAndTerminateLink(link);
- closeAndTerminateLink(this.link);
- setLink(null);
- cancelTasks();
- serverResource.disconnected();
- logger.info("Lost connection to host: {}. Attempting reconnection while we still have {} commands in progress.", shell.getConnectedHost(), commandsInProgress.get());
- stopAndCleanupConnection(true);
- String host = preferredMSHost;
- if (org.apache.commons.lang3.StringUtils.isBlank(host)) {
- host = shell.getNextHost();
- }
- List<String> avoidMSHostList = shell.getAvoidHosts();
- do {
- if (CollectionUtils.isEmpty(avoidMSHostList) || !avoidMSHostList.contains(host)) {
- connection = new NioClient(getAgentName(), host, shell.getPort(), shell.getWorkers(), shell.getSslHandshakeTimeout(), this);
- logger.info("Reconnecting to host: {}", host);
- try {
- connection.start();
- } catch (final NioConnectionException e) {
- logger.info("Attempted to re-connect to the server, but received an unexpected exception, trying again...", e);
- stopAndCleanupConnection(false);
- }
+ NioConnection connection = this.connection;
+ connection.stop();
+ try {
+ connection.cleanUp();
+ } catch (final IOException e) {
+ logger.warn("Fail to clean up old connection", e);
+ }
+
+ try {
+ while (connection.isStartup()) {
+ logger.debug("Waiting for connection graceful stop");
+ shell.getBackoffAlgorithm().waitBeforeRetry();
+ connection.stop();
}
- shell.getBackoffAlgorithm().waitBeforeRetry();
- host = shell.getNextHost();
- } while (!connection.isStartup());
- shell.updateConnectedHost(((NioClient)connection).getHost());
- logger.info("Connected to the host: {}", shell.getConnectedHost());
+ } catch (Exception e) {
+ logger.warn("Failed to gracefully stop connection", e);
+ }
+ logger.debug("Connection stopped");
+ }
+
+ /**
+ * Select the host to reconnect to based on priority:
+ * 1. preferredHost if defined and not blank
+ * 2. Link's socket address IP if available and not null
+ * 3. shell.getNextHost() if the above two options are not met
+ *
+ * @param preferredHost the preferred host to connect to
+ * @param link the current link which may contain socket address information
+ * @return the host to connect to
+ */
+ protected String selectReconnectionHost(String preferredHost, Link link) {
+ return Optional.ofNullable(preferredHost)
+ .filter(org.apache.commons.lang3.StringUtils::isNotBlank)
+ .orElseGet(() -> Optional.ofNullable(link)
+ .map(Link::getSocketAddress)
+ .map(InetSocketAddress::getAddress)
+ .map(InetAddress::getHostAddress)
+ .orElseGet(shell::getNextHost));
}
- protected void closeAndTerminateLink(final Link link) {
- if (link == null) {
+ /**
+ * Reconnect to Management Server.
+ *
+ * @param link - connection holder
+ * @param preferredHost - if defined, reconnect will be performed to this Host first,
+ * otherwise will be used {@link IAgentShell#getNextHost()}
+ * @param forceReconnect - expected to be true if called by {@link MigrateAgentConnectionCommand},
+ * this is only "switch Management Server", it does not perform full Host Connect process.
+ */
+ protected void reconnect(Link link, String preferredHost, boolean forceReconnect) {
+ if (!reconnectLock.compareAndSet(false, true)) {
+ logger.warn("Reconnect is already running, exiting");
return;
}
- link.close();
- link.terminated();
+ String requestedLink = Optional.ofNullable(link).map(Link::toString).orElse("N/A");
+ String currentLink = Optional.ofNullable(this.link).map(Link::toString).orElse("N/A");
+ logger.info("Reconnect info: provided link: {}, agent link: {}, preferred host: {}, force" +
+ " reconnect: {}", requestedLink, currentLink, preferredHost, forceReconnect);
+
+ try {
+ logger.debug("Obtained reconnect lock");
+ if (!(forceReconnect || reconnectAllowed)) {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Reconnect requested but it is not allowed {}", link);
+ }
+ return;
+ }
+
+ if (isReconnectStormDetected(link, preferredHost, requestedLink, currentLink)) {
+ return;
+ }
+
+ cleanupConnectionBeforeReconnect(link);
+ // start with preferred host
+ String host = selectReconnectionHost(preferredHost, link);
+
+ String hostLog = LogUtils.getHostLog(host, shell.getPort());
+ List<String> avoidMsHostList = Optional.ofNullable(shell.getAvoidHosts()).orElseGet(List::of);
+ // pointer to the first element of "refuse loop"
+ AtomicReference<String> firstRefuseLoopHostRef = new AtomicReference<>(null);
+ // to break deadlock where "non-avoid" MS Hosts are down and only "avoid" are up
+ AtomicBoolean ignoreAvoidMsHostListRef = new AtomicBoolean(false);
+ do {
+ AtomicBoolean skipTimeoutRef = new AtomicBoolean(false);
+ String parentLogContextId = (String) ThreadContext.get("logcontextid");
+ if (parentLogContextId != null) {
+ ThreadContext.put("logcontextid-parent", parentLogContextId);
+ }
+ ThreadContext.put("logcontextid", UuidUtils.first(UUID.randomUUID().toString()));
+ if (ignoreAvoidMsHostListRef.get() || !avoidMsHostList.contains(host)) {
+ connection = new NioClient(getAgentName(), host, shell.getPort(), shell.getWorkers(),
+ shell.getSslHandshakeTimeout(), this);
+ logger.info("Reconnecting to host: {}", hostLog);
+ try {
+ connection.start();
+ // successfully connected, skip the rest
+ continue;
+ } catch (Exception e) {
+ logReconnectionFailure(e, hostLog);
+
+ try {
+ stopAndCleanupConnection();
+ } catch (Exception ex) {
+ logger.warn("Got an exception during stop and cleanup connection", e);
+ }
+
+ updateRefuseLoopState(e, host, firstRefuseLoopHostRef, ignoreAvoidMsHostListRef, skipTimeoutRef);
+ }
+ } else {
+ logger.debug("Next host {} is in avoid list, skipped", hostLog);
+ if (org.apache.commons.lang3.StringUtils.isBlank(preferredHost)) {
+ logHostLists(avoidMsHostList);
+ skipTimeoutRef.set(true);
+ }
+ }
+ if (!skipTimeoutRef.get()) {
+ shell.getBackoffAlgorithm().waitBeforeRetry();
+ }
+ host = shell.getNextHost();
+ hostLog = LogUtils.getHostLog(host, shell.getPort());
+ logger.debug("Next host to connect: {}", hostLog);
+ } while (!connection.isStartup());
+ // successfully connected
+ shell.updateConnectedHost(((NioClient) connection).getHost());
+ String msg = String.format("Connected to the host: %s (%s)", shell.getConnectedHost(), this.link);
+ logger.info(msg);
+ } finally {
+ reconnectLock.set(false);
+ logger.debug("Removed reconnect lock");
+ }
}
- protected void stopAndCleanupConnection(boolean waitForStop) {
- if (connection == null) {
- return;
+ /**
+ * Handles "Connection refused" loop detection and determines if backoff timeout should be skipped.
+ * Manages refuse loop state to detect when all management servers have been tried and need
+ * to ignore avoid list to prevent deadlock.
+ *
+ * @param e the exception from connection attempt
+ * @param host the current host being attempted
+ * @param firstRefuseLoopHostRef reference to first host in refuse loop (modified by this method)
+ * @param ignoreAvoidMsHostListRef flag to ignore avoid list (modified by this method)
+ * @return true if timeout should be skipped (connection refused), false otherwise
+ */
+ private void updateRefuseLoopState(Exception e, String host, AtomicReference<String> firstRefuseLoopHostRef, AtomicBoolean ignoreAvoidMsHostListRef, AtomicBoolean skipTimeoutRef) {
+ // we are skipping timeout for "Connection refused" to not waste time on down MS
+ boolean skipTimeout = Optional.ofNullable(e.getCause())
+ .filter(ConnectException.class::isInstance)
+ .map(Throwable::getMessage)
+ .filter(CONNECTION_REFUSED_MSG::equalsIgnoreCase)
+ .isPresent();
+ skipTimeoutRef.set(skipTimeout);
+ String firstRefuseLoopHost = firstRefuseLoopHostRef.get();
+ // for each "Connection refused" (maybe need to have a copy of variable with better name)
+ // start "refuse loop"
+ if (skipTimeout && firstRefuseLoopHost == null) {
+ firstRefuseLoopHostRef.set(host);
+ ignoreAvoidMsHostListRef.set(false);
+ logger.debug("Started refuse loop for host {}", firstRefuseLoopHost);
+ // closed "refuse loop"
+ } else if (skipTimeout && firstRefuseLoopHost.equalsIgnoreCase(host)) {
+ ignoreAvoidMsHostListRef.set(true);
+ logger.debug("Closed refuse loop for host {}", firstRefuseLoopHost);
+ // got non "refuse" related issue, break "refuse loop"
+ } else if (!skipTimeout && (firstRefuseLoopHostRef != null || ignoreAvoidMsHostListRef.get())) {
+ logger.debug("Broke refuse loop for host {} by {}", firstRefuseLoopHost, host);
+ firstRefuseLoopHostRef.set(null);
+ ignoreAvoidMsHostListRef.set(false);
}
- connection.stop();
+ }
+
+ /**
+ * Logs reconnection failure with appropriate level based on rejection reason.
+ * If connection was rejected due to max concurrent connections limit (Broken pipe),
+ * logs as warning. Otherwise logs as info.
+ *
+ * @param e the exception that occurred during reconnection attempt
+ * @param hostLog the formatted host log string for logging
+ */
+ private void logReconnectionFailure(Exception e, String hostLog) {
+ // check if got NIO Connection exception, caused by IO Exception "Broken pipe"
+ boolean rejectedByMs = Optional.of(e).filter(NioConnectionException.class::isInstance)
+ .map(Exception::getCause)
+ .filter(IOException.class::isInstance)
+ .map(IOException.class::cast)
+ .map(IOException::getMessage)
+ .filter(BROKEN_PIPE_MSG::equalsIgnoreCase)
+ .isPresent();
+ if (rejectedByMs) {
+ logger.warn("Attempted to re-connect to {}, but rejected" +
+ " due to 'agent.max.concurrent.new.connections' reached limit," +
+ " will try again", hostLog, e);
+ } else {
+ logger.info("Attempted to re-connect to {}, but got exception," +
+ " will try again", hostLog, e);
+ }
+ }
+
+ /**
+ * Logs all management server host lists for debugging reconnection logic.
+ * Outputs defined hosts, hosts to avoid, and calculated available hosts.
+ *
+ * @param avoidMsHostList list of management server hosts to avoid during reconnection
+ */
+ private void logHostLists(List<String> avoidMsHostList) {
+ logger.debug("Preferred host is not defined");
try {
- connection.cleanUp();
- } catch (final IOException e) {
- logger.warn("Fail to clean up old connection. {}", e);
+ List<String> hostsList = Optional.ofNullable(shell.getHosts())
+ .map(Arrays::asList)
+ .orElseGet(List::of);
+
+ List<String> hostsShortList = new ArrayList<>(hostsList);
+ hostsShortList.removeAll(avoidMsHostList);
+
+ logger.info("Defined hosts: {} Avoid hosts: {} Available hosts: {}",
+ String.join(", ", hostsList), String.join(", ", avoidMsHostList), String.join(", ", hostsShortList));
+ } catch (Exception e) {
+ logger.warn("Failed to calculate next host logic", e);
}
- if (!waitForStop) {
- return;
+ }
+
+ /**
+ * Cleans up current connection state before attempting reconnection.
+ * Stops host connect process, terminates links, cancels scheduled tasks,
+ * notifies server resource about disconnection, and resets connection tracking.
+ *
+ * @param link the link that triggered reconnection
+ */
+ private void cleanupConnectionBeforeReconnect(Link link) {
+ String lastConnectedHost = shell.getConnectedHost();
+ try {
+ // reset Host status track and Startup process initiating
+ logger.debug("Stopping Host Connect process");
+ hostConnectProcess.stop();
+ closeAndTerminateLink(link);
+ closeAndTerminateLink(this.link);
+ setLink(null);
+ cancelTasks();
+ serverResource.disconnected();
+ stopAndCleanupConnection();
+ shell.updateConnectedHost(null);
+ } catch (Exception ex) {
+ logger.error("Failed to cleanup previous connection", ex);
+ }
+ logger.info("Lost connection to host: {}. Attempting reconnection while we still have" +
+ " {} commands in progress.", lastConnectedHost, commandsInProgress.get());
+ }
+
+ /**
+ * Detects reconnection storm by checking if the reconnect request is redundant.
+ * This prevents processing stale reconnection requests for old links when
+ * agent has already established a new connection.
+ *
+ * @param link the link requesting reconnection
+ * @param preferredHost the preferred host to reconnect to (may be null)
+ * @param requestedLink string representation of the requested link for logging
+ * @param currentLink string representation of the current agent link for logging
+ * @return true if reconnection storm is detected and request should be skipped, false otherwise
+ */
+ private boolean isReconnectStormDetected(Link link, String preferredHost, String requestedLink, String currentLink) {
+ logger.debug("Calling storm guard");
+ boolean reconnectForCurrentLink = link == this.link;
+ boolean currentLinkTerminated = this.link != null && this.link.isTerminated();
+ boolean reconnectForNewHost = this.hostname != null && this.hostname.equals(preferredHost);
+ // if none of the above is true
Review Comment:
`reconnectForNewHost` compares `preferredHost` to `this.hostname` (the agent’s own hostname), which is unrelated to the management server host selection and can cause the storm-guard to behave incorrectly. Consider basing this on the currently connected management server host instead (or drop this condition entirely if link/termination checks are sufficient).
##########
server/src/main/java/org/apache/cloudstack/agent/lb/IndirectAgentLBServiceImpl.java:
##########
@@ -446,6 +446,7 @@ protected boolean migrateNonRoutingHostAgentsInZone(String fromMsUuid, long from
break;
}
+ // FIXME: it is fire and forget, Management Server will never know if task failed
migrateAgentsExecutorService.submit(new MigrateAgentConnectionTask(fromMsId, hostId, dc.getId(), orderedHostIdList, avoidMsList, lbCheckInterval, lbAlgorithm, lbAlgorithmChanged));
}
Review Comment:
There’s an explicit `FIXME` noting the migration tasks are fire-and-forget and failures are not observable by the management server. Leaving this as-is can lead to silent partial migrations. Consider tracking submitted futures and reporting failures (or propagating them via status/metrics) instead of ignoring task outcomes.
##########
agent/src/main/java/com/cloud/agent/Agent.java:
##########
@@ -623,98 +559,363 @@ public Task create(final Task.Type type, final Link link, final byte[] data) {
return new ServerHandler(type, link, data);
}
- protected void reconnect(final Link link) {
- reconnect(link, null, false);
+ protected void closeAndTerminateLink(Link link) {
+ Optional.ofNullable(link)
+ .map(Link::attachment)
+ .filter(ServerAttache.class::isInstance)
+ .map(ServerAttache.class::cast)
+ .ifPresentOrElse(ServerAttache::disconnect, () -> {
+ if (link != null) {
+ link.close();
+ link.terminated();
+ }
+ });
}
- protected void reconnect(final Link link, String preferredMSHost, boolean forTransfer) {
- if (!(forTransfer || reconnectAllowed)) {
- logger.debug("Reconnect requested but it is not allowed {}", () -> getLinkLog(link));
+ protected void stopAndCleanupConnection() {
+ if (connection == null) {
return;
}
- cancelStartupTask();
- closeAndTerminateLink(link);
- closeAndTerminateLink(this.link);
- setLink(null);
- cancelTasks();
- serverResource.disconnected();
- logger.info("Lost connection to host: {}. Attempting reconnection while we still have {} commands in progress.", shell.getConnectedHost(), commandsInProgress.get());
- stopAndCleanupConnection(true);
- String host = preferredMSHost;
- if (org.apache.commons.lang3.StringUtils.isBlank(host)) {
- host = shell.getNextHost();
- }
- List<String> avoidMSHostList = shell.getAvoidHosts();
- do {
- if (CollectionUtils.isEmpty(avoidMSHostList) || !avoidMSHostList.contains(host)) {
- connection = new NioClient(getAgentName(), host, shell.getPort(), shell.getWorkers(), shell.getSslHandshakeTimeout(), this);
- logger.info("Reconnecting to host: {}", host);
- try {
- connection.start();
- } catch (final NioConnectionException e) {
- logger.info("Attempted to re-connect to the server, but received an unexpected exception, trying again...", e);
- stopAndCleanupConnection(false);
- }
+ NioConnection connection = this.connection;
+ connection.stop();
+ try {
+ connection.cleanUp();
+ } catch (final IOException e) {
+ logger.warn("Fail to clean up old connection", e);
+ }
+
+ try {
+ while (connection.isStartup()) {
+ logger.debug("Waiting for connection graceful stop");
+ shell.getBackoffAlgorithm().waitBeforeRetry();
+ connection.stop();
}
- shell.getBackoffAlgorithm().waitBeforeRetry();
- host = shell.getNextHost();
- } while (!connection.isStartup());
- shell.updateConnectedHost(((NioClient)connection).getHost());
- logger.info("Connected to the host: {}", shell.getConnectedHost());
+ } catch (Exception e) {
+ logger.warn("Failed to gracefully stop connection", e);
+ }
+ logger.debug("Connection stopped");
+ }
+
+ /**
+ * Select the host to reconnect to based on priority:
+ * 1. preferredHost if defined and not blank
+ * 2. Link's socket address IP if available and not null
+ * 3. shell.getNextHost() if the above two options are not met
+ *
+ * @param preferredHost the preferred host to connect to
+ * @param link the current link which may contain socket address information
+ * @return the host to connect to
+ */
+ protected String selectReconnectionHost(String preferredHost, Link link) {
+ return Optional.ofNullable(preferredHost)
+ .filter(org.apache.commons.lang3.StringUtils::isNotBlank)
+ .orElseGet(() -> Optional.ofNullable(link)
+ .map(Link::getSocketAddress)
+ .map(InetSocketAddress::getAddress)
+ .map(InetAddress::getHostAddress)
+ .orElseGet(shell::getNextHost));
}
- protected void closeAndTerminateLink(final Link link) {
- if (link == null) {
+ /**
+ * Reconnect to Management Server.
+ *
+ * @param link - connection holder
+ * @param preferredHost - if defined, reconnect will be performed to this Host first,
+ * otherwise will be used {@link IAgentShell#getNextHost()}
+ * @param forceReconnect - expected to be true if called by {@link MigrateAgentConnectionCommand},
+ * this is only "switch Management Server", it does not perform full Host Connect process.
+ */
+ protected void reconnect(Link link, String preferredHost, boolean forceReconnect) {
+ if (!reconnectLock.compareAndSet(false, true)) {
+ logger.warn("Reconnect is already running, exiting");
return;
}
- link.close();
- link.terminated();
+ String requestedLink = Optional.ofNullable(link).map(Link::toString).orElse("N/A");
+ String currentLink = Optional.ofNullable(this.link).map(Link::toString).orElse("N/A");
+ logger.info("Reconnect info: provided link: {}, agent link: {}, preferred host: {}, force" +
+ " reconnect: {}", requestedLink, currentLink, preferredHost, forceReconnect);
+
+ try {
+ logger.debug("Obtained reconnect lock");
+ if (!(forceReconnect || reconnectAllowed)) {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Reconnect requested but it is not allowed {}", link);
+ }
+ return;
+ }
+
+ if (isReconnectStormDetected(link, preferredHost, requestedLink, currentLink)) {
+ return;
+ }
+
+ cleanupConnectionBeforeReconnect(link);
+ // start with preferred host
+ String host = selectReconnectionHost(preferredHost, link);
+
+ String hostLog = LogUtils.getHostLog(host, shell.getPort());
+ List<String> avoidMsHostList = Optional.ofNullable(shell.getAvoidHosts()).orElseGet(List::of);
+ // pointer to the first element of "refuse loop"
+ AtomicReference<String> firstRefuseLoopHostRef = new AtomicReference<>(null);
+ // to break deadlock where "non-avoid" MS Hosts are down and only "avoid" are up
+ AtomicBoolean ignoreAvoidMsHostListRef = new AtomicBoolean(false);
+ do {
+ AtomicBoolean skipTimeoutRef = new AtomicBoolean(false);
+ String parentLogContextId = (String) ThreadContext.get("logcontextid");
+ if (parentLogContextId != null) {
+ ThreadContext.put("logcontextid-parent", parentLogContextId);
+ }
+ ThreadContext.put("logcontextid", UuidUtils.first(UUID.randomUUID().toString()));
+ if (ignoreAvoidMsHostListRef.get() || !avoidMsHostList.contains(host)) {
+ connection = new NioClient(getAgentName(), host, shell.getPort(), shell.getWorkers(),
+ shell.getSslHandshakeTimeout(), this);
+ logger.info("Reconnecting to host: {}", hostLog);
+ try {
+ connection.start();
+ // successfully connected, skip the rest
+ continue;
+ } catch (Exception e) {
+ logReconnectionFailure(e, hostLog);
+
+ try {
+ stopAndCleanupConnection();
+ } catch (Exception ex) {
+ logger.warn("Got an exception during stop and cleanup connection", e);
Review Comment:
The catch block logs the wrong exception variable: it catches `ex` but logs `e`, which will hide the actual failure during cleanup and can mislead troubleshooting.
##########
framework/cluster/src/main/java/com/cloud/cluster/ClusterServiceServletHttpHandler.java:
##########
@@ -113,6 +117,14 @@ private void writeResponse(HttpResponse response, int statusCode, String content
response.setEntity(body);
}
+ private void logRequest(HttpRequest request, String requestBody) {
+ Optional<HttpRequest> requestOpt = Optional.ofNullable(request);
+ Optional<RequestLine> requestLineOpt = requestOpt.map(HttpRequest::getRequestLine);
+ String method = requestLineOpt.map(RequestLine::getMethod).orElse(null);
+ String uri = requestLineOpt.map(RequestLine::getUri).orElse(null);
+ logger.debug("{} {} {}", method, uri, requestBody);
+ }
Review Comment:
This debug log prints the full raw request body (all key/value parameters). Cluster PDUs can be large and may include sensitive data; logging the entire payload can create operational/security risk. Consider logging only method/URI plus size (or redact selected keys).
##########
agent/src/main/java/com/cloud/agent/AgentShell.java:
##########
@@ -95,6 +95,16 @@ public BackoffAlgorithm getBackoffAlgorithm() {
return _backoff;
}
+ @Override
+ public void setBackoffAlgorithm(BackoffAlgorithm backoffAlgorithm) {
+ this._backoff = backoffAlgorithm;
+ try {
+ backoffAlgorithm.getConfiguration().forEach((key, value) -> setPersistentProperty(null, key, value));
+ } catch (RuntimeException e) {
+ LOGGER.warn("Failed to persist backoff properties");
+ }
Review Comment:
The caught exception is swallowed in this log statement. Include the exception so failures to persist backoff settings are diagnosable.
##########
server/src/main/java/org/apache/cloudstack/agent/lb/IndirectAgentLBServiceImpl.java:
##########
@@ -595,7 +597,9 @@ protected void runInContext() {
msList = getManagementServerList(hostId, dcId, orderedHostIdList, lbAlgorithm);
}
+ // ask Host to reconnect to another Management Server
final MigrateAgentConnectionCommand cmd = new MigrateAgentConnectionCommand(msList, avoidMsList, lbAlgorithm, lbCheckInterval);
+ // timeout 1 minute (FIXME: should it be configurable?)
cmd.setWait(60);
final Answer answer = agentManager.easySend(hostId, cmd); //may not receive answer when the agent disconnects immediately and try reconnecting to other ms host
Review Comment:
The migration command timeout is hard-coded to 60 seconds and marked as a `FIXME`. This contradicts the PR goal of parameterized timeouts and makes tuning difficult in different environments. Consider using an existing config key or introducing one for this wait value.
##########
core/src/main/java/com/cloud/agent/api/AgentConnectStatusAnswer.java:
##########
@@ -0,0 +1,65 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package com.cloud.agent.api;
+
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.utils.db.GlobalLock;
+
+/**
+ * Answer for {@link AgentConnectStatusCommand}.
+ *
+ * @author mprokopchuk
+ */
+public class AgentConnectStatusAnswer extends Answer {
+
+ /**
+ * {@link Boolean#TRUE} means host has {@link GlobalLock#lock(int)} acquired, otherwise {@link Boolean#FALSE},
+ * and null if there is an error during executing {@link AgentConnectStatusCommand}.
+ */
Review Comment:
The `lockAvailable` Javadoc contradicts the field name and the consuming logic (e.g., HostConnectProcess treats TRUE as “lock is free/available”). Please clarify the meaning to avoid misinterpretation by future maintainers.
##########
agent/src/main/java/com/cloud/agent/Agent.java:
##########
@@ -623,98 +559,363 @@ public Task create(final Task.Type type, final Link link, final byte[] data) {
return new ServerHandler(type, link, data);
}
- protected void reconnect(final Link link) {
- reconnect(link, null, false);
+ protected void closeAndTerminateLink(Link link) {
+ Optional.ofNullable(link)
+ .map(Link::attachment)
+ .filter(ServerAttache.class::isInstance)
+ .map(ServerAttache.class::cast)
+ .ifPresentOrElse(ServerAttache::disconnect, () -> {
+ if (link != null) {
+ link.close();
+ link.terminated();
+ }
+ });
}
- protected void reconnect(final Link link, String preferredMSHost, boolean forTransfer) {
- if (!(forTransfer || reconnectAllowed)) {
- logger.debug("Reconnect requested but it is not allowed {}", () -> getLinkLog(link));
+ protected void stopAndCleanupConnection() {
+ if (connection == null) {
return;
}
- cancelStartupTask();
- closeAndTerminateLink(link);
- closeAndTerminateLink(this.link);
- setLink(null);
- cancelTasks();
- serverResource.disconnected();
- logger.info("Lost connection to host: {}. Attempting reconnection while we still have {} commands in progress.", shell.getConnectedHost(), commandsInProgress.get());
- stopAndCleanupConnection(true);
- String host = preferredMSHost;
- if (org.apache.commons.lang3.StringUtils.isBlank(host)) {
- host = shell.getNextHost();
- }
- List<String> avoidMSHostList = shell.getAvoidHosts();
- do {
- if (CollectionUtils.isEmpty(avoidMSHostList) || !avoidMSHostList.contains(host)) {
- connection = new NioClient(getAgentName(), host, shell.getPort(), shell.getWorkers(), shell.getSslHandshakeTimeout(), this);
- logger.info("Reconnecting to host: {}", host);
- try {
- connection.start();
- } catch (final NioConnectionException e) {
- logger.info("Attempted to re-connect to the server, but received an unexpected exception, trying again...", e);
- stopAndCleanupConnection(false);
- }
+ NioConnection connection = this.connection;
+ connection.stop();
+ try {
+ connection.cleanUp();
+ } catch (final IOException e) {
+ logger.warn("Fail to clean up old connection", e);
+ }
+
+ try {
+ while (connection.isStartup()) {
+ logger.debug("Waiting for connection graceful stop");
+ shell.getBackoffAlgorithm().waitBeforeRetry();
+ connection.stop();
}
- shell.getBackoffAlgorithm().waitBeforeRetry();
- host = shell.getNextHost();
- } while (!connection.isStartup());
- shell.updateConnectedHost(((NioClient)connection).getHost());
- logger.info("Connected to the host: {}", shell.getConnectedHost());
+ } catch (Exception e) {
+ logger.warn("Failed to gracefully stop connection", e);
+ }
+ logger.debug("Connection stopped");
+ }
+
+ /**
+ * Select the host to reconnect to based on priority:
+ * 1. preferredHost if defined and not blank
+ * 2. Link's socket address IP if available and not null
+ * 3. shell.getNextHost() if the above two options are not met
+ *
+ * @param preferredHost the preferred host to connect to
+ * @param link the current link which may contain socket address information
+ * @return the host to connect to
+ */
+ protected String selectReconnectionHost(String preferredHost, Link link) {
+ return Optional.ofNullable(preferredHost)
+ .filter(org.apache.commons.lang3.StringUtils::isNotBlank)
+ .orElseGet(() -> Optional.ofNullable(link)
+ .map(Link::getSocketAddress)
+ .map(InetSocketAddress::getAddress)
+ .map(InetAddress::getHostAddress)
+ .orElseGet(shell::getNextHost));
}
- protected void closeAndTerminateLink(final Link link) {
- if (link == null) {
+ /**
+ * Reconnect to Management Server.
+ *
+ * @param link - connection holder
+ * @param preferredHost - if defined, reconnect will be performed to this Host first,
+ * otherwise will be used {@link IAgentShell#getNextHost()}
+ * @param forceReconnect - expected to be true if called by {@link MigrateAgentConnectionCommand},
+ * this is only "switch Management Server", it does not perform full Host Connect process.
+ */
+ protected void reconnect(Link link, String preferredHost, boolean forceReconnect) {
+ if (!reconnectLock.compareAndSet(false, true)) {
+ logger.warn("Reconnect is already running, exiting");
return;
}
- link.close();
- link.terminated();
+ String requestedLink = Optional.ofNullable(link).map(Link::toString).orElse("N/A");
+ String currentLink = Optional.ofNullable(this.link).map(Link::toString).orElse("N/A");
+ logger.info("Reconnect info: provided link: {}, agent link: {}, preferred host: {}, force" +
+ " reconnect: {}", requestedLink, currentLink, preferredHost, forceReconnect);
+
+ try {
+ logger.debug("Obtained reconnect lock");
+ if (!(forceReconnect || reconnectAllowed)) {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Reconnect requested but it is not allowed {}", link);
+ }
+ return;
+ }
+
+ if (isReconnectStormDetected(link, preferredHost, requestedLink, currentLink)) {
+ return;
+ }
+
+ cleanupConnectionBeforeReconnect(link);
+ // start with preferred host
+ String host = selectReconnectionHost(preferredHost, link);
+
+ String hostLog = LogUtils.getHostLog(host, shell.getPort());
+ List<String> avoidMsHostList = Optional.ofNullable(shell.getAvoidHosts()).orElseGet(List::of);
+ // pointer to the first element of "refuse loop"
+ AtomicReference<String> firstRefuseLoopHostRef = new AtomicReference<>(null);
+ // to break deadlock where "non-avoid" MS Hosts are down and only "avoid" are up
+ AtomicBoolean ignoreAvoidMsHostListRef = new AtomicBoolean(false);
+ do {
+ AtomicBoolean skipTimeoutRef = new AtomicBoolean(false);
+ String parentLogContextId = (String) ThreadContext.get("logcontextid");
+ if (parentLogContextId != null) {
+ ThreadContext.put("logcontextid-parent", parentLogContextId);
+ }
+ ThreadContext.put("logcontextid", UuidUtils.first(UUID.randomUUID().toString()));
+ if (ignoreAvoidMsHostListRef.get() || !avoidMsHostList.contains(host)) {
+ connection = new NioClient(getAgentName(), host, shell.getPort(), shell.getWorkers(),
+ shell.getSslHandshakeTimeout(), this);
+ logger.info("Reconnecting to host: {}", hostLog);
+ try {
+ connection.start();
+ // successfully connected, skip the rest
+ continue;
+ } catch (Exception e) {
+ logReconnectionFailure(e, hostLog);
+
+ try {
+ stopAndCleanupConnection();
+ } catch (Exception ex) {
+ logger.warn("Got an exception during stop and cleanup connection", e);
+ }
+
+ updateRefuseLoopState(e, host, firstRefuseLoopHostRef, ignoreAvoidMsHostListRef, skipTimeoutRef);
+ }
+ } else {
+ logger.debug("Next host {} is in avoid list, skipped", hostLog);
+ if (org.apache.commons.lang3.StringUtils.isBlank(preferredHost)) {
+ logHostLists(avoidMsHostList);
+ skipTimeoutRef.set(true);
+ }
+ }
+ if (!skipTimeoutRef.get()) {
+ shell.getBackoffAlgorithm().waitBeforeRetry();
+ }
+ host = shell.getNextHost();
+ hostLog = LogUtils.getHostLog(host, shell.getPort());
+ logger.debug("Next host to connect: {}", hostLog);
+ } while (!connection.isStartup());
+ // successfully connected
+ shell.updateConnectedHost(((NioClient) connection).getHost());
+ String msg = String.format("Connected to the host: %s (%s)", shell.getConnectedHost(), this.link);
+ logger.info(msg);
+ } finally {
+ reconnectLock.set(false);
+ logger.debug("Removed reconnect lock");
+ }
}
- protected void stopAndCleanupConnection(boolean waitForStop) {
- if (connection == null) {
- return;
+ /**
+ * Handles "Connection refused" loop detection and determines if backoff timeout should be skipped.
+ * Manages refuse loop state to detect when all management servers have been tried and need
+ * to ignore avoid list to prevent deadlock.
+ *
+ * @param e the exception from connection attempt
+ * @param host the current host being attempted
+ * @param firstRefuseLoopHostRef reference to first host in refuse loop (modified by this method)
+ * @param ignoreAvoidMsHostListRef flag to ignore avoid list (modified by this method)
+ * @return true if timeout should be skipped (connection refused), false otherwise
+ */
+ private void updateRefuseLoopState(Exception e, String host, AtomicReference<String> firstRefuseLoopHostRef, AtomicBoolean ignoreAvoidMsHostListRef, AtomicBoolean skipTimeoutRef) {
+ // we are skipping timeout for "Connection refused" to not waste time on down MS
+ boolean skipTimeout = Optional.ofNullable(e.getCause())
+ .filter(ConnectException.class::isInstance)
+ .map(Throwable::getMessage)
+ .filter(CONNECTION_REFUSED_MSG::equalsIgnoreCase)
+ .isPresent();
+ skipTimeoutRef.set(skipTimeout);
+ String firstRefuseLoopHost = firstRefuseLoopHostRef.get();
+ // for each "Connection refused" (maybe need to have a copy of variable with better name)
+ // start "refuse loop"
+ if (skipTimeout && firstRefuseLoopHost == null) {
+ firstRefuseLoopHostRef.set(host);
+ ignoreAvoidMsHostListRef.set(false);
+ logger.debug("Started refuse loop for host {}", firstRefuseLoopHost);
Review Comment:
This log line uses `firstRefuseLoopHost` which is still null in this branch (it’s read before `firstRefuseLoopHostRef.set(host)`), so the message will always log `null`. Log the current `host` instead.
This issue also appears on line 742 of the same file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "Damans227 (via GitHub)" <gi...@apache.org>.
Damans227 commented on code in PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#discussion_r4162281815
##########
engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java:
##########
@@ -1253,15 +1635,16 @@ public Answer easySend(final Long hostId, final Command cmd) {
}
return answer;
-
+ // FIXME: There are a lot of result != null checks in callers. Should this method trow exception?
Review Comment:
ok thats gone now, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-5620310799
This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-6040334316
@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-6388945475
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19714
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-6466679667
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-5911594487
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "Damans227 (via GitHub)" <gi...@apache.org>.
Damans227 commented on code in PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#discussion_r4158490107
##########
agent/src/main/java/com/cloud/agent/HostConnectProcess.java:
##########
@@ -0,0 +1,355 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent;
+
+import com.cloud.agent.api.AgentConnectStatusAnswer;
+import com.cloud.agent.api.AgentConnectStatusCommand;
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.Command;
+import com.cloud.agent.api.StartupAnswer;
+import com.cloud.agent.api.StartupCommand;
+import com.cloud.agent.properties.AgentProperties;
+import com.cloud.agent.properties.AgentPropertiesFileHandler;
+import com.cloud.agent.transport.Request;
+import com.cloud.exception.CloudException;
+import com.cloud.exception.OperationTimedoutException;
+import com.cloud.host.Status;
+import com.cloud.resource.ResourceStatusUpdater;
+import com.cloud.resource.ServerResource;
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.cloud.utils.nio.Link;
+import org.apache.cloudstack.threadcontext.ThreadContextUtil;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.ThreadContext;
+
+import java.io.IOException;
+import java.nio.channels.ClosedChannelException;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Predicate;
+
+public class HostConnectProcess {
+ private static final Logger logger = LogManager.getLogger(HostConnectProcess.class);
+
+ public static final int DEFAULT_ASYNC_COMMAND_TIMEOUT_SEC =
+ AgentPropertiesFileHandler.getPropertyValue(AgentProperties.ASYNC_COMMAND_TIMEOUT_SEC);
+
+ public static final int DEFAULT_ASYNC_STARTUP_COMMAND_TIMEOUT_SEC =
+ AgentPropertiesFileHandler.getPropertyValue(AgentProperties.ASYNC_STARTUP_COMMAND_TIMEOUT_SEC);
+
+ static final long HOST_STATUS_CHECK_INITIAL_DELAY_SEC = 10;
+ private long hostStatusCheckDelaySec = AgentPropertiesFileHandler.getPropertyValue(AgentProperties.AGENT_HOST_STATUS_CHECK_DELAY_SEC);
+ private final AtomicReference<ScheduledFuture<?>> hostStatusFutureRef = new AtomicReference<>();
+ private final Agent agent;
+ private ScheduledExecutorService hostStatusExecutor;
+
+ public HostConnectProcess(Agent agent) {
+ this.agent = agent;
+ initExecutors();
+ }
+
+ private void initExecutors() {
+ stop();
+ var threadFactory = new NamedThreadFactory("Agent-" + HostStatusTask.class.getSimpleName());
+ hostStatusExecutor = Executors.newScheduledThreadPool(1, threadFactory);
+ }
+
+ /**
+ * Stops the whole connect process and cancels all scheduled asynchronous tasks.
+ * Returns {@link Boolean#TRUE} if {@link HostConnectProcess} was waiting for {@link StartupAnswer}.
+ */
+ public boolean stop() {
+ logger.debug("Stopping connect process. The process is active: {}", isInProgress());
+ stopHostStatusExecutor();
+ logger.debug("Stopped executor");
+ Optional<? extends ScheduledFuture<?>> hostStatusOpt = Optional.ofNullable(hostStatusFutureRef.getAndSet(null))
+ .filter(Predicate.not(ScheduledFuture::isCancelled));
+
+ hostStatusOpt.ifPresent(future -> future.cancel(true));
+ logger.debug("Cancelled future");
+
+ return hostStatusOpt.isPresent();
+ }
+
+ private void stopHostStatusExecutor() {
+ if (hostStatusExecutor != null) {
+ hostStatusExecutor.shutdownNow();
+ hostStatusExecutor = null;
+ }
+ }
+
+ public void scheduleConnectProcess(Link link, boolean connectionTransfer) {
+ logger.debug("Scheduling connect process for {}", link);
+ initExecutors();
+
+ var task = new HostStatusTask(link, connectionTransfer, agent, hostStatusFutureRef);
+ var future = hostStatusExecutor.scheduleWithFixedDelay(ThreadContextUtil.wrapThreadContext(task),
+ HOST_STATUS_CHECK_INITIAL_DELAY_SEC,
+ hostStatusCheckDelaySec, TimeUnit.SECONDS);
+ hostStatusFutureRef.set(future);
+ }
+
+ /**
+ * Returns {@link Boolean#TRUE} if {@link HostStatusTask} created and scheduled.
+ * That means there is already {@link Status#Connecting} process is running.
+ */
+ public boolean isInProgress() {
+ return Optional.ofNullable(hostStatusFutureRef.get())
+ .filter(Predicate.not(ScheduledFuture::isCancelled)).isPresent();
+ }
+
+ public void updateHostStatusCheckDelay(int newDelaySec) {
+ logger.info("Updating host status check delay from {} to {} seconds", hostStatusCheckDelaySec, newDelaySec);
+ this.hostStatusCheckDelaySec = newDelaySec;
+ }
+
+ /**
+ * Task wait for the Host to be available to connect to submit {@link StartupCommand}.
Review Comment:
got it, thanks
##########
agent/src/main/java/com/cloud/agent/HostConnectProcess.java:
##########
@@ -0,0 +1,355 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent;
+
+import com.cloud.agent.api.AgentConnectStatusAnswer;
+import com.cloud.agent.api.AgentConnectStatusCommand;
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.Command;
+import com.cloud.agent.api.StartupAnswer;
+import com.cloud.agent.api.StartupCommand;
+import com.cloud.agent.properties.AgentProperties;
+import com.cloud.agent.properties.AgentPropertiesFileHandler;
+import com.cloud.agent.transport.Request;
+import com.cloud.exception.CloudException;
+import com.cloud.exception.OperationTimedoutException;
+import com.cloud.host.Status;
+import com.cloud.resource.ResourceStatusUpdater;
+import com.cloud.resource.ServerResource;
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.cloud.utils.nio.Link;
+import org.apache.cloudstack.threadcontext.ThreadContextUtil;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.ThreadContext;
+
+import java.io.IOException;
+import java.nio.channels.ClosedChannelException;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Predicate;
+
+public class HostConnectProcess {
+ private static final Logger logger = LogManager.getLogger(HostConnectProcess.class);
+
+ public static final int DEFAULT_ASYNC_COMMAND_TIMEOUT_SEC =
+ AgentPropertiesFileHandler.getPropertyValue(AgentProperties.ASYNC_COMMAND_TIMEOUT_SEC);
+
+ public static final int DEFAULT_ASYNC_STARTUP_COMMAND_TIMEOUT_SEC =
+ AgentPropertiesFileHandler.getPropertyValue(AgentProperties.ASYNC_STARTUP_COMMAND_TIMEOUT_SEC);
+
+ static final long HOST_STATUS_CHECK_INITIAL_DELAY_SEC = 10;
+ private long hostStatusCheckDelaySec = AgentPropertiesFileHandler.getPropertyValue(AgentProperties.AGENT_HOST_STATUS_CHECK_DELAY_SEC);
Review Comment:
ok thats consistent now, thanks
##########
utils/src/main/java/org/apache/cloudstack/threadcontext/ThreadContextUtil.java:
##########
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.threadcontext;
+
+import org.apache.logging.log4j.ThreadContext;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Utility class, helps to propagate {@link ThreadContext} values from parent to child threads.
+ *
+ * @author mprokopchuk
+ */
+public class ThreadContextUtil {
+ /**
+ * Wrap {@link Runnable} to propagate {@link ThreadContext} values.
+ *
+ * @param delegate
+ * @return
+ */
+ public static Runnable wrapThreadContext(Runnable delegate) {
+ @SuppressWarnings("unchecked")
Review Comment:
ok that covers it, thanks
##########
engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java:
##########
@@ -762,40 +985,124 @@ public void notifyMonitorsOfNewlyAddedHost(long hostId) {
protected AgentAttache notifyMonitorsOfConnection(final AgentAttache attache, final StartupCommand[] cmds, final boolean forRebalance) throws ConnectionException {
final long hostId = attache.getId();
final HostVO host = _hostDao.findById(hostId);
- for (final Pair<Integer, Listener> monitor : _hostMonitors) {
- logger.debug("Sending Connect to listener: {}, for rebalance: {}", monitor.second().getClass().getSimpleName(), forRebalance);
- for (StartupCommand cmd : cmds) {
- try {
- logger.debug("process connection to issue: {} for host: {}, forRebalance: {}", ReflectionToStringBuilderUtils.reflectOnlySelectedFields(cmd, "id", "type", "msHostList", "connectionTransferred"), hostId, forRebalance);
- monitor.second().processConnect(host, cmd, forRebalance);
- } catch (final ConnectionException ce) {
- if (ce.isSetupError()) {
- logger.warn("Monitor {} says there is an error in the connect process for {} due to {}", monitor.second().getClass().getSimpleName(), hostId, ce.getMessage());
- handleDisconnectWithoutInvestigation(attache, Event.AgentDisconnected, true, true);
- throw ce;
- } else {
- logger.info("Monitor {} says not to continue the connect process for {} due to {}", monitor.second().getClass().getSimpleName(), hostId, ce.getMessage());
- handleDisconnectWithoutInvestigation(attache, Event.ShutdownRequested, true, true);
- return attache;
- }
- } catch (final HypervisorVersionChangedException hvce) {
- handleDisconnectWithoutInvestigation(attache, Event.ShutdownRequested, true, true);
- throw new CloudRuntimeException("Unable to connect " + attache.getId(), hvce);
- } catch (final Exception e) {
- logger.error("Monitor {} says there is an error in the connect process for {} due to {}", monitor.second().getClass().getSimpleName(), hostId, e.getMessage(), e);
- handleDisconnectWithoutInvestigation(attache, Event.AgentDisconnected, true, true);
- throw new CloudRuntimeException("Unable to connect " + attache.getId(), e);
- }
+ Optional<HostVO> hostOpt = Optional.ofNullable(host);
+ String hostName = hostOpt.map(HostVO::getName).orElse(null);
+ String hostUuid = hostOpt.map(HostVO::getUuid).orElse(null);
+ boolean processSuccessful;
+ Profiler processProfiler = new Profiler();
+ processProfiler.start();
+ logger.debug("Process connect for host: {} ({}) for rebalance: {}", hostName, hostUuid, forRebalance);
+ try {
+ processSuccessful = performProcessConnect(attache, cmds, forRebalance, hostName, hostUuid, host);
+ } finally {
+ processProfiler.stop();
+ long processDurationSec = processProfiler.getDurationInMillis() / 1000;
+ logger.debug("Finished process connect for host: {} ({}) for rebalance: {} duration, s: {}",
+ hostName, hostUuid, forRebalance, processDurationSec);
+ }
+ if (processSuccessful) {
+ sendReadyCommand(attache, host, hostId);
+
+ agentStatusTransitTo(host, Event.Ready, _nodeId);
+ attache.ready();
+ }
+ return attache;
+ }
+
+ /**
+ * @return true if commands successfully completed
+ */
+ private boolean performProcessConnect(AgentAttache attache, StartupCommand[] cmds, boolean forRebalance, String hostName, String hostUuid, HostVO host) throws ConnectionException {
+ int monitorsSize = _hostMonitors.size();
+ boolean processSuccessful = true;
+ for (int monitorIndex = 0; monitorIndex < monitorsSize && processSuccessful; monitorIndex++) {
+ Pair<Integer, Listener> monitor = _hostMonitors.get(monitorIndex);
+ Listener listener = monitor.second();
+ String monitorClassName = listener.getClass().getSimpleName();
+ Profiler monitorProfiler = new Profiler();
+ logger.debug("Process connect for host: {} ({}) listener {} ({} of {}) for rebalance: {}",
+ hostName, hostUuid, monitorClassName, monitorIndex, monitorsSize, forRebalance);
+ try {
+ monitorProfiler.start();
+ processSuccessful = performProcessConnect(attache, cmds, forRebalance, hostName, hostUuid, monitorClassName, listener, host);
+ } finally {
+ monitorProfiler.stop();
+ long monitorDurationSec = monitorProfiler.getDurationInMillis() / 1000;
+ logger.debug("Finished process connect for host: {} ({}) listener: {} ({} of {}) for rebalance: {} duration, s: {}",
+ hostName, hostUuid, monitorClassName, monitorIndex, monitorsSize, forRebalance, monitorDurationSec);
+ }
+ }
+ return processSuccessful;
+ }
+
+ /**
+ * @return true if commands successfully completed
+ */
+ private boolean performProcessConnect(AgentAttache attache, StartupCommand[] cmds, boolean forRebalance, String hostName, String hostUuid, String monitorClassName, Listener listener, HostVO host) throws ConnectionException {
+ boolean processSuccessful = true;
+ int commandsSize = cmds.length;
+ for (int commandIndex = 0; commandIndex < commandsSize && processSuccessful; commandIndex++) {
+ StartupCommand command = cmds[commandIndex];
+ String commandClassName = command.getClass().getSimpleName();
+ boolean connTransferred = command.isConnectionTransferred();
+ long processStart = System.currentTimeMillis();
+ try {
+ logger.debug("Process command: {} ({} of {}) in connect for host: {} ({}) listener: {} for rebalance: {} connection transferred: {}",
+ commandClassName, commandIndex, commandsSize, hostName, hostUuid, monitorClassName, forRebalance, connTransferred);
+ listener.processConnect(host, command, forRebalance);
+ } catch (Exception e) {
+ handleConnectionException(attache, e, processStart, host, monitorClassName);
+ processSuccessful = false;
+ } finally {
+ long commandDurationSec = (System.currentTimeMillis() - processStart) / 1000;
+ logger.debug("Finished processing command: {} ({} of {}) in connect for host: {} ({}) listener: {} for rebalance: {} connection transferred: {} duration, s: {}",
+ commandClassName, commandIndex, commandsSize, hostName, hostUuid, monitorClassName, forRebalance, connTransferred, commandDurationSec);
+ }
+ }
+ return processSuccessful;
+ }
+
+ private void handleConnectionException(AgentAttache attache, Exception e, long processStart, HostVO host, String monitorClassName) throws ConnectionException {
+ long processDurationMs = System.currentTimeMillis() - processStart;
+ StringBuilder summaryBuilder = getSummaryMsgBuilder("Failed to connect Host",
+ EventTypes.EVENT_HOST_RECONNECT, host.getUuid(), host.getName(), null,
+ e.getMessage(), monitorClassName, processDurationMs);
+ logger.fatal(summaryBuilder.toString(), e);
+
+ // add alert to the Host here
+ if (e instanceof ConnectionException) {
+ ConnectionException ce = (ConnectionException) e;
+ // XXX: in case of Storage Pool issue we are ending up here
Review Comment:
got it, thanks
##########
framework/db/src/main/java/com/cloud/utils/db/GlobalLock.java:
##########
@@ -45,127 +43,248 @@
* </p>
*/
public class GlobalLock {
- protected Logger logger = LogManager.getLogger(getClass());
+ protected final static Logger logger = LogManager.getLogger(GlobalLock.class);
private String name;
- private int lockCount = 0;
- private Thread ownerThread = null;
-
- private int referenceCount = 0;
- private long holdingStartTick = 0;
-
- private static Map<String, GlobalLock> s_lockMap = new HashMap<String, GlobalLock>();
+ /**
+ * DB lock count.
+ * Increments on {@link GlobalLock#lock(int)} and decrements on {@link GlobalLock#unlock()}.
+ * Upon {@link GlobalLock#unlock()}, if {@link GlobalLock#lockCount} is less than 1, then lock removed from DB
+ */
+ private int lockCount;
+
+ /**
+ * Internal (in-memory) lock count.
+ * Increments on {@link GlobalLock#addRef()} and indirectly on {@link GlobalLock#getInternLock(String)} and
+ * decrements on {@link GlobalLock#releaseRef()}, {@link GlobalLock#unlock()} and on {@link GlobalLock#lock(int)}
+ * if DB lock is unsuccessful
+ */
+ private int referenceCount;
+
+ /**
+ * Thread that owns lock. If lock called from different thread, it will be waiting for the owner to unlock it
+ * within requested timeout. If owner thread call {@link GlobalLock#lock(int)} again, then
+ * {@link GlobalLock#lockCount} will be incremented.
+ * If {@link GlobalLock#unlock()} called by owner thread, or DB lock will be unsuccessful, then owner thread will be
+ * nullified.
+ */
+ private Thread ownerThread;
+
+ /**
+ * Variable to hold lock duration in milliseconds. Used for information only.
+ */
+ private long holdingStartTick;
+
+ /**
+ * Holds all created locks.
+ */
+ private static Map<String, GlobalLock> s_lockMap = new HashMap<>();
+
+ /**
+ * Create lock.
+ *
+ * @param name lock name
+ */
private GlobalLock(String name) {
this.name = name;
}
+ /**
+ * Increment reference count to lock.
+ *
+ * @return reference count
+ */
public int addRef() {
synchronized (this) {
referenceCount++;
return referenceCount;
}
}
+ /**
+ * Decrement reference count to lock.
+ *
+ * @return reference count
+ */
public int releaseRef() {
- int refCount;
-
boolean needToRemove = false;
synchronized (this) {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Releasing reference for internal lock {}, reference count: {}, lock count: {}",
+ name, referenceCount, lockCount);
+ }
referenceCount--;
- refCount = referenceCount;
-
- if (referenceCount < 0)
- logger.warn("Unmatched Global lock " + name + " reference usage detected, check your code!");
- if (referenceCount == 0)
+ if (referenceCount < 0) {
+ logger.warn("Unmatched internal lock {} reference usage detected (reference count: {}, " +
+ "lock count: {}), check your code!", name, referenceCount, lockCount);
+ } else if (referenceCount < 1) {
needToRemove = true;
+ }
}
- if (needToRemove)
+ if (needToRemove) {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Need to release internal lock {}", name);
+ }
releaseInternLock(name);
+ }
+ if (logger.isDebugEnabled()) {
+ logger.debug("Released reference for lock {}, reference count: {}", name, referenceCount);
+ }
+ return referenceCount;
+ }
- return refCount;
+ public static boolean isLockAvailable(String name) {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Checking lock availability for {}", name);
+ }
+ boolean result = false;
+ try {
+ result = DbUtil.isFreeLock(name);
+ } finally {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Result of checking lock availability for {}: {}", name, result);
+ }
+ }
+ return result;
}
+ /**
+ * Registers internal lock (in memory) object. Does not create any lock in DB yet.
+ *
+ * @param name lock name
+ * @return lock object
+ */
public static GlobalLock getInternLock(String name) {
synchronized (s_lockMap) {
+ GlobalLock lock;
if (s_lockMap.containsKey(name)) {
- GlobalLock lock = s_lockMap.get(name);
- lock.addRef();
- return lock;
+ lock = s_lockMap.get(name);
+ if (logger.isDebugEnabled()) {
+ logger.debug("Internal lock {} already exists with reference count {} and lock count {}",
+ name, lock.referenceCount, lock.lockCount);
+ }
} else {
- GlobalLock lock = new GlobalLock(name);
- lock.addRef();
+ lock = new GlobalLock(name);
+ if (logger.isDebugEnabled()) {
+ logger.debug("Internal lock {} does not exist, adding", name);
+ }
s_lockMap.put(name, lock);
- return lock;
}
+ lock.addRef();
+ if (logger.isDebugEnabled()) {
+ logger.debug("Added reference to internal lock {}, reference count {}, lock count {}",
+ name, lock.referenceCount, lock.lockCount);
+ }
+ return lock;
}
}
+ /**
+ * Unregister internal lock (in memory) object. Does not remove any lock from DB.
+ *
+ * @param name lock name
+ */
private void releaseInternLock(String name) {
synchronized (s_lockMap) {
GlobalLock lock = s_lockMap.get(name);
if (lock != null) {
- if (lock.referenceCount == 0)
+ if (lock.referenceCount == 0) {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Released internal lock {}", name);
+ }
s_lockMap.remove(name);
+ } else {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Not releasing internal lock {} as it has references count: {}, lock count: {}",
+ name, lock.referenceCount, lock.lockCount);
+ }
+ }
} else {
- logger.warn("Releasing " + name + ", but it is already released.");
+ logger.warn("Internal lock {} already released", name);
}
}
}
+ /**
+ * Acquire or join existing DB lock.
+ *
+ * @param timeoutSeconds time in seconds during which lock needs to be obtained (it is not the lock duration)
+ * @return true if lock successfully obtained
+ */
public boolean lock(int timeoutSeconds) {
int remainingMilliSeconds = timeoutSeconds * 1000;
Profiler profiler = new Profiler();
boolean interrupted = false;
try {
while (true) {
synchronized (this) {
- if (ownerThread != null && ownerThread == Thread.currentThread()) {
- logger.warn("Global lock re-entrance detected");
-
+ if (ownerThread == Thread.currentThread()) {
+ logger.warn("Global lock {} re-entrance detected, owner thread: {}, reference count: {}, " +
+ "lock count: {}", name, getThreadName(ownerThread), referenceCount, lockCount);
+ // if it is re-entrance, then we may have more lock counts than needed?
lockCount++;
- if (logger.isTraceEnabled())
- logger.trace("lock " + name + " is acquired, lock count :" + lockCount);
+ if (logger.isDebugEnabled()) {
+ logger.debug("Global lock {} joined, reference count: {}, lock count: {}",
+ name, referenceCount, lockCount);
+ }
return true;
- }
-
- if (ownerThread != null) {
+ } else if (ownerThread != null) {
profiler.start();
try {
- wait((timeoutSeconds) * 1000L);
+ logger.debug("Waiting {} seconds to acquire global lock {}", timeoutSeconds, name);
+ wait(timeoutSeconds * 1000L);
} catch (InterruptedException e) {
interrupted = true;
}
profiler.stop();
remainingMilliSeconds -= profiler.getDurationInMillis();
- if (remainingMilliSeconds < 0)
+ if (remainingMilliSeconds < 0) {
+ logger.warn("Timeout of {} seconds to acquire global lock {} has been reached, " +
+ "owner thread {}, reference count: {}, lock count: {}", timeoutSeconds, name, getThreadName(ownerThread), referenceCount, lockCount);
return false;
+ }
continue;
} else {
// take ownership temporarily to prevent others enter into stage of acquiring DB lock
ownerThread = Thread.currentThread();
+ // XXX: do we need it here (???)
Review Comment:
got it, thanks
##########
server/src/main/java/org/apache/cloudstack/agent/lb/IndirectAgentLBServiceImpl.java:
##########
@@ -446,6 +446,7 @@ protected boolean migrateNonRoutingHostAgentsInZone(String fromMsUuid, long from
break;
}
+ // FIXME: it is fire and forget, Management Server will never know if task failed
Review Comment:
ok thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "nvazquez (via GitHub)" <gi...@apache.org>.
nvazquez commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-6158384353
@blueorangutan test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-5729957973
Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 19282
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-4649855176
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18190
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-6464521415
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#discussion_r4398978404
##########
agent/src/main/java/com/cloud/agent/HostConnectProcess.java:
##########
@@ -0,0 +1,347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent;
+
+import com.cloud.agent.api.AgentConnectStatusAnswer;
+import com.cloud.agent.api.AgentConnectStatusCommand;
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.Command;
+import com.cloud.agent.api.StartupAnswer;
+import com.cloud.agent.api.StartupCommand;
+import com.cloud.agent.properties.AgentProperties;
+import com.cloud.agent.properties.AgentPropertiesFileHandler;
+import com.cloud.agent.transport.Request;
+import com.cloud.exception.CloudException;
+import com.cloud.exception.OperationTimedoutException;
+import com.cloud.host.Status;
+import com.cloud.resource.ResourceStatusUpdater;
+import com.cloud.resource.ServerResource;
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.cloud.utils.nio.Link;
+import org.apache.cloudstack.threadcontext.ThreadContextCommandUtil;
+import org.apache.cloudstack.threadcontext.ThreadContextUtil;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.LogManager;
+
+import java.io.IOException;
+import java.nio.channels.ClosedChannelException;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Predicate;
+
+public class HostConnectProcess {
+ private static final Logger logger = LogManager.getLogger(HostConnectProcess.class);
+
+ public static final int DEFAULT_ASYNC_COMMAND_TIMEOUT_SEC =
+ AgentPropertiesFileHandler.getPropertyValue(AgentProperties.ASYNC_COMMAND_TIMEOUT_SEC);
+
+ public static final int DEFAULT_ASYNC_STARTUP_COMMAND_TIMEOUT_SEC =
+ AgentPropertiesFileHandler.getPropertyValue(AgentProperties.ASYNC_STARTUP_COMMAND_TIMEOUT_SEC);
+
+ static final long HOST_STATUS_CHECK_INITIAL_DELAY_SEC = 10;
+ private long hostStatusCheckDelaySec = AgentPropertiesFileHandler.getPropertyValue(AgentProperties.AGENT_HOST_STATUS_CHECK_DELAY_SEC);
+ private final AtomicReference<ScheduledFuture<?>> hostStatusFutureRef = new AtomicReference<>();
+ private final Agent agent;
+ private ScheduledExecutorService hostStatusExecutor;
+
+ public HostConnectProcess(Agent agent) {
+ this.agent = agent;
+ initExecutors();
+ }
+
+ private void initExecutors() {
+ stop();
+ var threadFactory = new NamedThreadFactory("Agent-" + HostStatusTask.class.getSimpleName());
+ hostStatusExecutor = Executors.newScheduledThreadPool(1, threadFactory);
Review Comment:
This class creates a non-daemon scheduled executor, but `Agent.stop()` does not stop the `HostConnectProcess`. If the agent is shut down while a connection is pending, the status-check thread and its scheduled task remain alive and can continue issuing requests/reconnects after the agent's request handler and connection have been torn down. Add an explicit lifecycle call to stop this process during agent shutdown (and ensure the executor is closed).
This issue also appears on line 193 of the same file.
##########
utils/src/main/java/com/cloud/utils/backoff/BackoffAlgorithm.java:
##########
@@ -35,4 +37,5 @@ public interface BackoffAlgorithm extends Adapter {
*/
void reset();
+ Map<String, String> getConfiguration();
Review Comment:
Adding this abstract method makes the public `BackoffAlgorithm` extension point source- and binary-incompatible for custom backoff implementations (including classes selected through `backoff.implementation`). Such an implementation can still be instantiated by `BackoffFactory`, but `AgentShell.setBackoffAlgorithm()` will fail when it invokes the newly required method. Preserve compatibility with a default configuration method or otherwise handle implementations compiled against the previous interface.
##########
agent/src/main/java/com/cloud/agent/SynchronousListener.java:
##########
@@ -0,0 +1,102 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent;
+
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.Command;
+import com.cloud.utils.Profiler;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+/**
+ * Blocks the thread and waits for the answer.
+ *
+ * @author mprokopchuk
+ */
+public class SynchronousListener implements ServerListener {
+ private static final Logger logger = LogManager.getLogger(SynchronousListener.class);
+
+ protected Answer[] _answers;
+ protected boolean _disconnected;
+ private int _timeout;
+
+ public synchronized Answer[] getAnswers() {
+ return _answers;
+ }
+
+ @Override
+ public boolean isRecurring() {
+ return false;
+ }
+
+ public synchronized boolean isDisconnected() {
+ return _disconnected;
+ }
+
+ @Override
+ public synchronized boolean processAnswers(long seq, Answer[] resp) {
+ _answers = resp;
+ notifyAll();
+ return true;
+ }
+
+ @Override
+ public boolean processCommands(long seq, Command[] commands) {
+ return false;
+ }
+
+ @Override
+ public synchronized boolean processDisconnect() {
+ logger.trace("Server disconnected. Will notify waiters");
+ _disconnected = true;
+ notifyAll();
+ return true;
+ }
+
+ public synchronized Answer[] waitFor(int s) throws InterruptedException {
+ if (_disconnected) {
+ return null;
+ }
+
+ if (_answers != null) {
+ return _answers;
+ }
+
+ Profiler profiler = new Profiler();
+ profiler.start();
+ if (s <= 0) {
+ wait();
+ } else {
+ int ms = s * 1000;
+ wait(ms);
+ }
+ profiler.stop();
+
+ logger.trace("Synchronized command - sending completed, time: {}, answer: {}",
+ profiler.getDurationInMillis(), _answers != null ? _answers[0].toString() : "null");
Review Comment:
`ServerAttache.send` explicitly accepts and checks empty answer arrays, but this log expression dereferences element zero whenever `_answers` is non-null. An empty response (or a response with a null first entry) therefore throws `ArrayIndexOutOfBoundsException`/`NullPointerException` before the caller can handle the empty response. Guard both the array length and the element before calling `toString()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "Copilot (via GitHub) via commits" <co...@cloudstack.apache.org>.
Copilot commented on code in PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#discussion_r4457630901
##########
utils/src/main/java/org/apache/cloudstack/threadcontext/ThreadContextUtil.java:
##########
@@ -0,0 +1,120 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.threadcontext;
+
+import com.cloud.utils.StringUtils;
+import com.google.gson.Gson;
+import com.google.gson.reflect.TypeToken;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.ThreadContext;
+
+import java.lang.reflect.Type;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Utility class, helps to propagate {@link ThreadContext} values from parent to child threads.
+ *
+ * @author mprokopchuk
+ */
+public class ThreadContextUtil {
+ private static final Logger logger = LogManager.getLogger(ThreadContextUtil.class);
+
+ public static final String MDC_UUID_KEY = "uuid";
+ public static final String MDC_LOG_CONTEXT_ID_KEY = "logcontextid";
+ public static final String CONTEXT_UUID_KEY = "uuid";
+ public static final String CONTEXT_LOG_ID_KEY = "logid";
+
+ /**
+ * Wrap {@link Runnable} to propagate {@link ThreadContext} values.
+ *
+ * @param delegate the runnable to execute with the calling thread's {@link ThreadContext} values
+ * @return a runnable that installs the captured context, runs the delegate and restores the previous context
+ */
+ public static Runnable wrapThreadContext(Runnable delegate) {
+ Map<String, String> context = ThreadContext.getContext() != null ?
+ new HashMap<>(ThreadContext.getContext()) : null;
+
+ return () -> {
+ Map<String, String> oldContext = ThreadContext.getContext() != null ?
+ new HashMap<>(ThreadContext.getContext()) : null;
+ try {
+ ThreadContext.clearMap();
+ if (context != null) {
+ context.forEach(ThreadContext::put);
+ }
+ delegate.run();
+ } finally {
+ ThreadContext.clearMap();
+ if (oldContext != null) {
+ oldContext.forEach(ThreadContext::put);
+ }
+ }
+ };
+ }
+
+ /**
+ * Set UUID in ThreadContext.
+ *
+ * @param uuid the UUID value to set
+ */
+ public static void setUuid(String uuid) {
+ if (StringUtils.isNotEmpty(uuid)) {
+ ThreadContext.put(MDC_UUID_KEY, uuid);
+ }
Review Comment:
This leaves the previous UUID in Log4j's thread-local context when `uuid` is blank. `ApiDispatcher` calls this for every request, and the API worker threads are reused without a corresponding ThreadContext clear, so a request without a UUID can inherit and propagate the UUID from the preceding request to async jobs and agent commands. Remove the MDC key when the argument is empty, as `ThreadContextCommandUtil.putOrRemove` does.
##########
engine/schema/src/main/resources/META-INF/db/schema-42210to42300.sql:
##########
@@ -651,3 +651,16 @@ WHERE `name`='user.vm.readonly.details' AND `value` IS NOT NULL;
-- usage records introduced in 4.22.1 (cumulative and per-VM) can coexist. See #13399.
CALL `cloud_usage`.`IDEMPOTENT_DROP_INDEX`('id', 'cloud_usage.usage_volume');
CALL `cloud_usage`.`IDEMPOTENT_ADD_UNIQUE_INDEX`('cloud_usage.usage_volume', 'id', '(volume_id ASC, created ASC, vm_id ASC)');
+
+-- these indexes created from top 20 resource consuming DB queries
+CALL `cloud`.`IDEMPOTENT_ADD_INDEX`('idx_mshost_removed_state','cloud.mshost', '(removed, state)');
+CALL `cloud`.`IDEMPOTENT_ADD_INDEX`('idx_host_id_removed','cloud.host', '(id, removed)');
+CALL `cloud`.`IDEMPOTENT_ADD_INDEX`('idx_networks_id_removed','cloud.networks', '(id, removed)');
+CALL `cloud`.`IDEMPOTENT_ADD_INDEX`('idx_alert_type_data_center_id_archived_pod_id','cloud.alert', '(type, data_center_id, archived, pod_id)');
+CALL `cloud`.`IDEMPOTENT_ADD_INDEX`('idx_configuration_name','cloud.configuration', '(name)');
Review Comment:
`cloud.configuration` already has the `i_configuration__name` index added by `schema-228to229.sql:62`. Adding another single-column index on the same column under a new name provides no query benefit but increases storage and write cost for every configuration change. Please remove this duplicate index (or replace the existing one rather than adding a second copy).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-4619614226
## [Codecov](https://app.codecov.io/gh/apache/cloudstack/pull/13345?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 3.51%. Comparing base ([`be51948`](https://app.codecov.io/gh/apache/cloudstack/commit/be5194814616ec8d3bf9410ed6bc533186bd0eed?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`ad7a8c5`](https://app.codecov.io/gh/apache/cloudstack/commit/ad7a8c576ef10afcf3c8176e2ca3058ca3f6e29d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
> :exclamation: There is a different number of reports uploaded between BASE (be51948) and HEAD (ad7a8c5). Click for more details.
>
> <details><summary>HEAD has 1 upload less than BASE</summary>
>
>| Flag | BASE (be51948) | HEAD (ad7a8c5) |
>|------|------|------|
>|unittests|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #13345 +/- ##
=============================================
- Coverage 18.10% 3.51% -14.60%
=============================================
Files 6037 464 -5573
Lines 542798 40159 -502639
Branches 66457 7560 -58897
=============================================
- Hits 98298 1413 -96885
+ Misses 433453 38556 -394897
+ Partials 11047 190 -10857
```
| [Flag](https://app.codecov.io/gh/apache/cloudstack/pull/13345/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [uitests](https://app.codecov.io/gh/apache/cloudstack/pull/13345/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `3.51% <ø> (ø)` | |
| [unittests](https://app.codecov.io/gh/apache/cloudstack/pull/13345/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/cloudstack/pull/13345?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on code in PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#discussion_r3373466837
##########
core/src/main/java/com/cloud/agent/api/AgentConnectStatusCommand.java:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent.api;
+
+/**
+ * Command to check status of {@link StartupCommand} from the Agent.
+ *
+ * @author mprokopchuk
+ */
Review Comment:
@bernardodemarco I think, it's ok to have it. we've author mentioned in other classes as well. (check with '@author' in the code base).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-4625626090
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18154
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-4625255628
@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-4690888245
This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-5925427522
@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-6388114548
@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#discussion_r4399329000
##########
agent/src/main/java/com/cloud/agent/HostConnectProcess.java:
##########
@@ -0,0 +1,347 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent;
+
+import com.cloud.agent.api.AgentConnectStatusAnswer;
+import com.cloud.agent.api.AgentConnectStatusCommand;
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.Command;
+import com.cloud.agent.api.StartupAnswer;
+import com.cloud.agent.api.StartupCommand;
+import com.cloud.agent.properties.AgentProperties;
+import com.cloud.agent.properties.AgentPropertiesFileHandler;
+import com.cloud.agent.transport.Request;
+import com.cloud.exception.CloudException;
+import com.cloud.exception.OperationTimedoutException;
+import com.cloud.host.Status;
+import com.cloud.resource.ResourceStatusUpdater;
+import com.cloud.resource.ServerResource;
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.cloud.utils.nio.Link;
+import org.apache.cloudstack.threadcontext.ThreadContextCommandUtil;
+import org.apache.cloudstack.threadcontext.ThreadContextUtil;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.LogManager;
+
+import java.io.IOException;
+import java.nio.channels.ClosedChannelException;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Predicate;
+
+public class HostConnectProcess {
+ private static final Logger logger = LogManager.getLogger(HostConnectProcess.class);
+
+ public static final int DEFAULT_ASYNC_COMMAND_TIMEOUT_SEC =
+ AgentPropertiesFileHandler.getPropertyValue(AgentProperties.ASYNC_COMMAND_TIMEOUT_SEC);
+
+ public static final int DEFAULT_ASYNC_STARTUP_COMMAND_TIMEOUT_SEC =
+ AgentPropertiesFileHandler.getPropertyValue(AgentProperties.ASYNC_STARTUP_COMMAND_TIMEOUT_SEC);
+
+ static final long HOST_STATUS_CHECK_INITIAL_DELAY_SEC = 10;
+ private long hostStatusCheckDelaySec = AgentPropertiesFileHandler.getPropertyValue(AgentProperties.AGENT_HOST_STATUS_CHECK_DELAY_SEC);
+ private final AtomicReference<ScheduledFuture<?>> hostStatusFutureRef = new AtomicReference<>();
+ private final Agent agent;
+ private ScheduledExecutorService hostStatusExecutor;
+
+ public HostConnectProcess(Agent agent) {
+ this.agent = agent;
+ initExecutors();
+ }
+
+ private void initExecutors() {
+ stop();
+ var threadFactory = new NamedThreadFactory("Agent-" + HostStatusTask.class.getSimpleName());
+ hostStatusExecutor = Executors.newScheduledThreadPool(1, threadFactory);
+ }
+
+ /**
+ * Stops the whole connect process and cancels all scheduled asynchronous tasks.
+ * Returns {@link Boolean#TRUE} if {@link HostConnectProcess} was waiting for {@link StartupAnswer}.
+ */
+ public boolean stop() {
+ logger.debug("Stopping connect process. The process is active: {}", isInProgress());
+ stopHostStatusExecutor();
+ logger.debug("Stopped executor");
+ Optional<? extends ScheduledFuture<?>> hostStatusOpt = Optional.ofNullable(hostStatusFutureRef.getAndSet(null))
+ .filter(Predicate.not(ScheduledFuture::isCancelled));
+
+ hostStatusOpt.ifPresent(future -> future.cancel(true));
+ logger.debug("Cancelled future");
+
+ return hostStatusOpt.isPresent();
+ }
+
+ private void stopHostStatusExecutor() {
+ if (hostStatusExecutor != null) {
+ hostStatusExecutor.shutdownNow();
+ hostStatusExecutor = null;
+ }
+ }
+
+ public void scheduleConnectProcess(Link link, boolean connectionTransfer) {
+ logger.debug("Scheduling connect process for {}", link);
+ initExecutors();
+
+ var task = new HostStatusTask(link, connectionTransfer, agent);
+ var future = hostStatusExecutor.scheduleWithFixedDelay(ThreadContextUtil.wrapThreadContext(task),
+ HOST_STATUS_CHECK_INITIAL_DELAY_SEC,
+ hostStatusCheckDelaySec, TimeUnit.SECONDS);
+ task.setFuture(future);
+ hostStatusFutureRef.set(future);
Review Comment:
This method can be entered concurrently from the initial CONNECT task and from a PingAnswer requesting startup. Each caller runs `initExecutors()` and then overwrites `hostStatusFutureRef`, so the future created by the first caller can continue running without a reference while `stop()` only cancels the second one. Both tasks can then observe an available lock and send duplicate StartupCommands, recreating the connection storm this class is intended to prevent. Serialize/reject concurrent scheduling (and atomically replace and cancel the prior generation).
This issue also appears in the following locations of the same file:
- line 191
- line 199
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-6468791168
@blueorangutan test keepEnv
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-6489568122
<b>[SF] Trillian test result (tid-17310)</b>
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 59401 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13345-t17310-kvm-ol8.zip
Smoke tests completed. 155 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_uservm_host_control_state | `Failure` | 17.11 | test_host_control_state.py
test_hostha_enable_ha_when_host_disabled | `Error` | 3.09 | test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenance | `Error` | 303.23 | test_hostha_kvm.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on code in PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#discussion_r3373466837
##########
core/src/main/java/com/cloud/agent/api/AgentConnectStatusCommand.java:
##########
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent.api;
+
+/**
+ * Command to check status of {@link StartupCommand} from the Agent.
+ *
+ * @author mprokopchuk
+ */
Review Comment:
@bernardodemarco I think, it's ok to have it. we've author mentioned in other classes as well. (check with '`@author`' in the code base).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "Damans227 (via GitHub)" <gi...@apache.org>.
Damans227 commented on code in PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#discussion_r3624822556
##########
agent/src/main/java/com/cloud/agent/ServerAttache.java:
##########
@@ -0,0 +1,489 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent;
+
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.Command;
+import com.cloud.agent.transport.Request;
+import com.cloud.agent.transport.Response;
+import com.cloud.exception.AgentUnavailableException;
+import com.cloud.exception.CloudException;
+import com.cloud.exception.OperationTimedoutException;
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.cloud.utils.nio.Link;
+import org.apache.cloudstack.managed.context.ManagedContextRunnable;
+import org.apache.cloudstack.utils.reflectiontostringbuilderutils.ReflectionToStringBuilderUtils;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.ThreadContext;
+
+import java.io.IOException;
+import java.nio.channels.ClosedChannelException;
+import java.security.SecureRandom;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * ServerAttache provides basic server communication commands to be implemented.
+ *
+ * @author mprokopchuk
Review Comment:
Nit: leftover personal `@author mprokopchuk` javadoc tag (recurs in several of the other new files here too: SynchronousListener.java, AgentConnectStatusAnswer.java, AgentConnectStatusCommand.java, ThreadContextUtil.java, ConfigKeyUtil.java, BackoffFactory.java, ExponentialWithJitterBackoff.java). Probably copied over from wherever this was originally authored, worth stripping.
##########
agent/src/main/java/com/cloud/agent/HostConnectProcess.java:
##########
@@ -0,0 +1,355 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent;
+
+import com.cloud.agent.api.AgentConnectStatusAnswer;
+import com.cloud.agent.api.AgentConnectStatusCommand;
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.Command;
+import com.cloud.agent.api.StartupAnswer;
+import com.cloud.agent.api.StartupCommand;
+import com.cloud.agent.properties.AgentProperties;
+import com.cloud.agent.properties.AgentPropertiesFileHandler;
+import com.cloud.agent.transport.Request;
+import com.cloud.exception.CloudException;
+import com.cloud.exception.OperationTimedoutException;
+import com.cloud.host.Status;
+import com.cloud.resource.ResourceStatusUpdater;
+import com.cloud.resource.ServerResource;
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.cloud.utils.nio.Link;
+import org.apache.cloudstack.threadcontext.ThreadContextUtil;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.ThreadContext;
+
+import java.io.IOException;
+import java.nio.channels.ClosedChannelException;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Predicate;
+
+public class HostConnectProcess {
+ private static final Logger logger = LogManager.getLogger(HostConnectProcess.class);
+
+ public static final int DEFAULT_ASYNC_COMMAND_TIMEOUT_SEC =
+ AgentPropertiesFileHandler.getPropertyValue(AgentProperties.ASYNC_COMMAND_TIMEOUT_SEC);
+
+ public static final int DEFAULT_ASYNC_STARTUP_COMMAND_TIMEOUT_SEC =
+ AgentPropertiesFileHandler.getPropertyValue(AgentProperties.ASYNC_STARTUP_COMMAND_TIMEOUT_SEC);
+
+ static final long HOST_STATUS_CHECK_INITIAL_DELAY_SEC = 10;
+ private long hostStatusCheckDelaySec = AgentPropertiesFileHandler.getPropertyValue(AgentProperties.AGENT_HOST_STATUS_CHECK_DELAY_SEC);
Review Comment:
Nit: this class mixes underscore-prefixed fields (`_link`, `_futureRef`) with camelCase fields like this one in the same class.
##########
agent/src/main/java/com/cloud/agent/ServerAttache.java:
##########
@@ -0,0 +1,489 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent;
+
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.Command;
+import com.cloud.agent.transport.Request;
+import com.cloud.agent.transport.Response;
+import com.cloud.exception.AgentUnavailableException;
+import com.cloud.exception.CloudException;
+import com.cloud.exception.OperationTimedoutException;
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.cloud.utils.nio.Link;
+import org.apache.cloudstack.managed.context.ManagedContextRunnable;
+import org.apache.cloudstack.utils.reflectiontostringbuilderutils.ReflectionToStringBuilderUtils;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.ThreadContext;
+
+import java.io.IOException;
+import java.nio.channels.ClosedChannelException;
+import java.security.SecureRandom;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * ServerAttache provides basic server communication commands to be implemented.
+ *
+ * @author mprokopchuk
+ */
+public class ServerAttache {
+ private static final Logger logger = LogManager.getLogger(ServerAttache.class);
+
+ private static final ScheduledExecutorService s_listenerExecutor = Executors.newScheduledThreadPool(10,
+ new NamedThreadFactory("ListenerTimer"));
+
+ protected static Comparator<Request> s_reqComparator = (o1, o2) -> {
+ long seq1 = o1.getSequence();
+ long seq2 = o2.getSequence();
+ if (seq1 < seq2) {
+ return -1;
+ } else if (seq1 > seq2) {
+ return 1;
+ } else {
+ return 0;
+ }
+ };
+
+ protected static Comparator<Object> s_seqComparator = (o1, o2) -> {
+ long seq1 = ((Request) o1).getSequence();
+ long seq2 = (Long) o2;
+ if (seq1 < seq2) {
+ return -1;
+ } else if (seq1 > seq2) {
+ return 1;
+ } else {
+ return 0;
+ }
+ };
+
+ private static final Random s_rand = new SecureRandom();
+ protected String _name;
+ private Link _link;
+ protected ConcurrentHashMap<Long, ServerListener> _waitForList;
+ protected ConcurrentHashMap<Long, java.util.concurrent.ScheduledFuture<?>> _alarmFutures;
+ protected LinkedList<Request> _requests;
+ protected Long _currentSequence;
+ protected long _nextSequence;
+
+ protected ServerAttache(Link link) {
+ _name = link.getIpAddress();
+ _link = link;
+ _waitForList = new ConcurrentHashMap<>();
+ _alarmFutures = new ConcurrentHashMap<>();
+ _requests = new LinkedList<>();
+ _nextSequence = Long.valueOf(s_rand.nextInt(Short.MAX_VALUE)) << 48;
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ServerAttache %s", ReflectionToStringBuilderUtils.reflectOnlySelectedFields(this,
+ "_name"));
+ }
+
+ public synchronized long getNextSequence() {
+ return ++_nextSequence;
+ }
+
+ protected synchronized void addRequest(Request req) {
+ int index = findRequest(req);
+ assert (index < 0) : "How can we get index again? " + index + ":" + req.toString();
+ _requests.add(-index - 1, req);
+ }
+
+ protected void cancel(Request req) {
+ cancel(req.getSequence());
+ }
+
+ protected synchronized void cancel(long seq) {
+ logger.debug(log(seq, "Cancelling."));
+
+ ServerListener listener = _waitForList.remove(seq);
+ if (listener != null) {
+ listener.processDisconnect();
+ }
+ int index = findRequest(seq);
+ if (index >= 0) {
+ _requests.remove(index);
+ }
+ }
+
+ protected synchronized int findRequest(Request req) {
+ return Collections.binarySearch(_requests, req, s_reqComparator);
+ }
+
+ protected synchronized int findRequest(long seq) {
+ return Collections.binarySearch(_requests, seq, s_seqComparator);
+ }
+
+ protected String log(long seq, String msg) {
+ return "Seq " + _name + "-" + seq + ": " + msg;
+ }
+
+ protected void registerListener(long seq, ServerListener listener) {
+ if (logger.isTraceEnabled()) {
+ logger.trace(log(seq, "Registering listener"));
+ }
+ if (listener.getTimeout() != -1) {
+ java.util.concurrent.ScheduledFuture<?> alarmFuture =
+ s_listenerExecutor.schedule(new Alarm(seq), listener.getTimeout(), TimeUnit.SECONDS);
+ _alarmFutures.put(seq, alarmFuture);
+ }
+ _waitForList.put(seq, listener);
+ }
+
+ protected ServerListener unregisterListener(long sequence) {
+ if (logger.isTraceEnabled()) {
+ logger.trace(log(sequence, "Unregistering listener"));
+ }
+ java.util.concurrent.ScheduledFuture<?> alarmFuture = _alarmFutures.remove(sequence);
+ if (alarmFuture != null) {
+ alarmFuture.cancel(false);
+ }
+ return _waitForList.remove(sequence);
+ }
+
+ protected ServerListener getListener(long sequence) {
+ return _waitForList.get(sequence);
+ }
+
+ public String getName() {
+ return _name;
+ }
+
+ public int getQueueSize() {
+ return _requests.size();
+ }
+
+ public boolean processAnswers(long seq, Response resp) {
+ resp.logD("Processing: ", true);
+ boolean processed = false;
+ Answer[] answers = resp.getAnswers();
+ try {
+ ServerListener monitor = getListener(seq);
+ if (monitor == null) {
+ if (answers[0] != null && answers[0].getResult()) {
+ processed = true;
+ }
+ logger.debug(log(seq, "Unable to find listener."));
+ } else {
+ processed = monitor.processAnswers(seq, answers);
+ logger.trace(log(seq, (processed ? "" : " did not ") + " processed "));
Review Comment:
Nit: this produces "did not processed" (double space, missing verb) when not processed.
##########
agent/src/main/java/com/cloud/agent/AgentShell.java:
##########
@@ -568,6 +582,7 @@ public static void main(String[] args) {
shell.init(args);
shell.start();
} catch (ConfigurationException e) {
+ LOGGER.fatal(e.getMessage(), e);
Review Comment:
Nit: this now logs the same message twice, `LOGGER.fatal` here followed by `System.out.println` right below with the same `e.getMessage()`.
##########
agent/src/main/java/com/cloud/agent/HostConnectProcess.java:
##########
@@ -0,0 +1,355 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent;
+
+import com.cloud.agent.api.AgentConnectStatusAnswer;
+import com.cloud.agent.api.AgentConnectStatusCommand;
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.Command;
+import com.cloud.agent.api.StartupAnswer;
+import com.cloud.agent.api.StartupCommand;
+import com.cloud.agent.properties.AgentProperties;
+import com.cloud.agent.properties.AgentPropertiesFileHandler;
+import com.cloud.agent.transport.Request;
+import com.cloud.exception.CloudException;
+import com.cloud.exception.OperationTimedoutException;
+import com.cloud.host.Status;
+import com.cloud.resource.ResourceStatusUpdater;
+import com.cloud.resource.ServerResource;
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.cloud.utils.nio.Link;
+import org.apache.cloudstack.threadcontext.ThreadContextUtil;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.ThreadContext;
+
+import java.io.IOException;
+import java.nio.channels.ClosedChannelException;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Predicate;
+
+public class HostConnectProcess {
+ private static final Logger logger = LogManager.getLogger(HostConnectProcess.class);
+
+ public static final int DEFAULT_ASYNC_COMMAND_TIMEOUT_SEC =
+ AgentPropertiesFileHandler.getPropertyValue(AgentProperties.ASYNC_COMMAND_TIMEOUT_SEC);
+
+ public static final int DEFAULT_ASYNC_STARTUP_COMMAND_TIMEOUT_SEC =
+ AgentPropertiesFileHandler.getPropertyValue(AgentProperties.ASYNC_STARTUP_COMMAND_TIMEOUT_SEC);
+
+ static final long HOST_STATUS_CHECK_INITIAL_DELAY_SEC = 10;
+ private long hostStatusCheckDelaySec = AgentPropertiesFileHandler.getPropertyValue(AgentProperties.AGENT_HOST_STATUS_CHECK_DELAY_SEC);
+ private final AtomicReference<ScheduledFuture<?>> hostStatusFutureRef = new AtomicReference<>();
+ private final Agent agent;
+ private ScheduledExecutorService hostStatusExecutor;
+
+ public HostConnectProcess(Agent agent) {
+ this.agent = agent;
+ initExecutors();
+ }
+
+ private void initExecutors() {
+ stop();
+ var threadFactory = new NamedThreadFactory("Agent-" + HostStatusTask.class.getSimpleName());
+ hostStatusExecutor = Executors.newScheduledThreadPool(1, threadFactory);
+ }
+
+ /**
+ * Stops the whole connect process and cancels all scheduled asynchronous tasks.
+ * Returns {@link Boolean#TRUE} if {@link HostConnectProcess} was waiting for {@link StartupAnswer}.
+ */
+ public boolean stop() {
+ logger.debug("Stopping connect process. The process is active: {}", isInProgress());
+ stopHostStatusExecutor();
+ logger.debug("Stopped executor");
+ Optional<? extends ScheduledFuture<?>> hostStatusOpt = Optional.ofNullable(hostStatusFutureRef.getAndSet(null))
+ .filter(Predicate.not(ScheduledFuture::isCancelled));
+
+ hostStatusOpt.ifPresent(future -> future.cancel(true));
+ logger.debug("Cancelled future");
+
+ return hostStatusOpt.isPresent();
+ }
+
+ private void stopHostStatusExecutor() {
+ if (hostStatusExecutor != null) {
+ hostStatusExecutor.shutdownNow();
+ hostStatusExecutor = null;
+ }
+ }
+
+ public void scheduleConnectProcess(Link link, boolean connectionTransfer) {
+ logger.debug("Scheduling connect process for {}", link);
+ initExecutors();
+
+ var task = new HostStatusTask(link, connectionTransfer, agent, hostStatusFutureRef);
+ var future = hostStatusExecutor.scheduleWithFixedDelay(ThreadContextUtil.wrapThreadContext(task),
+ HOST_STATUS_CHECK_INITIAL_DELAY_SEC,
+ hostStatusCheckDelaySec, TimeUnit.SECONDS);
+ hostStatusFutureRef.set(future);
+ }
+
+ /**
+ * Returns {@link Boolean#TRUE} if {@link HostStatusTask} created and scheduled.
+ * That means there is already {@link Status#Connecting} process is running.
+ */
+ public boolean isInProgress() {
+ return Optional.ofNullable(hostStatusFutureRef.get())
+ .filter(Predicate.not(ScheduledFuture::isCancelled)).isPresent();
+ }
+
+ public void updateHostStatusCheckDelay(int newDelaySec) {
+ logger.info("Updating host status check delay from {} to {} seconds", hostStatusCheckDelaySec, newDelaySec);
+ this.hostStatusCheckDelaySec = newDelaySec;
+ }
+
+ /**
+ * Task wait for the Host to be available to connect to submit {@link StartupCommand}.
Review Comment:
Nit: javadoc typo, "Task wait for the Host" should be "waits".
##########
server/src/main/java/org/apache/cloudstack/agent/lb/IndirectAgentLBServiceImpl.java:
##########
@@ -446,6 +446,7 @@ protected boolean migrateNonRoutingHostAgentsInZone(String fromMsUuid, long from
break;
}
+ // FIXME: it is fire and forget, Management Server will never know if task failed
Review Comment:
Nit: new `FIXME` comment, might be worth a tracked follow-up instead of an inline note.
##########
engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java:
##########
@@ -1253,15 +1635,16 @@ public Answer easySend(final Long hostId, final Command cmd) {
}
return answer;
-
+ // FIXME: There are a lot of result != null checks in callers. Should this method trow exception?
Review Comment:
Nit: leftover `FIXME` comment carried over in the moved code.
##########
core/src/main/java/org/apache/cloudstack/threadcontext/ThreadContextUtil.java:
##########
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.threadcontext;
+
+import org.apache.logging.log4j.ThreadContext;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Utility class, helps to propagate {@link ThreadContext} values from parent to child threads.
+ *
+ * @author mprokopchuk
+ */
+public class ThreadContextUtil {
+ /**
+ * Wrap {@link Runnable} to propagate {@link ThreadContext} values.
+ *
+ * @param delegate
+ * @return
+ */
+ public static Runnable wrapThreadContext(Runnable delegate) {
+ @SuppressWarnings("unchecked")
Review Comment:
Nit: `@SuppressWarnings("unchecked")` here (and again below) looks unnecessary, no actual unchecked cast on this line since `ThreadContext.getContext()` is already generically typed. Also the javadoc above has empty `@param`/`@return` tags with no description.
##########
framework/db/src/main/java/com/cloud/utils/db/GlobalLock.java:
##########
@@ -45,127 +43,248 @@
* </p>
*/
public class GlobalLock {
- protected Logger logger = LogManager.getLogger(getClass());
+ protected final static Logger logger = LogManager.getLogger(GlobalLock.class);
private String name;
- private int lockCount = 0;
- private Thread ownerThread = null;
-
- private int referenceCount = 0;
- private long holdingStartTick = 0;
-
- private static Map<String, GlobalLock> s_lockMap = new HashMap<String, GlobalLock>();
+ /**
+ * DB lock count.
+ * Increments on {@link GlobalLock#lock(int)} and decrements on {@link GlobalLock#unlock()}.
+ * Upon {@link GlobalLock#unlock()}, if {@link GlobalLock#lockCount} is less than 1, then lock removed from DB
+ */
+ private int lockCount;
+
+ /**
+ * Internal (in-memory) lock count.
+ * Increments on {@link GlobalLock#addRef()} and indirectly on {@link GlobalLock#getInternLock(String)} and
+ * decrements on {@link GlobalLock#releaseRef()}, {@link GlobalLock#unlock()} and on {@link GlobalLock#lock(int)}
+ * if DB lock is unsuccessful
+ */
+ private int referenceCount;
+
+ /**
+ * Thread that owns lock. If lock called from different thread, it will be waiting for the owner to unlock it
+ * within requested timeout. If owner thread call {@link GlobalLock#lock(int)} again, then
+ * {@link GlobalLock#lockCount} will be incremented.
+ * If {@link GlobalLock#unlock()} called by owner thread, or DB lock will be unsuccessful, then owner thread will be
+ * nullified.
+ */
+ private Thread ownerThread;
+
+ /**
+ * Variable to hold lock duration in milliseconds. Used for information only.
+ */
+ private long holdingStartTick;
+
+ /**
+ * Holds all created locks.
+ */
+ private static Map<String, GlobalLock> s_lockMap = new HashMap<>();
+
+ /**
+ * Create lock.
+ *
+ * @param name lock name
+ */
private GlobalLock(String name) {
this.name = name;
}
+ /**
+ * Increment reference count to lock.
+ *
+ * @return reference count
+ */
public int addRef() {
synchronized (this) {
referenceCount++;
return referenceCount;
}
}
+ /**
+ * Decrement reference count to lock.
+ *
+ * @return reference count
+ */
public int releaseRef() {
- int refCount;
-
boolean needToRemove = false;
synchronized (this) {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Releasing reference for internal lock {}, reference count: {}, lock count: {}",
+ name, referenceCount, lockCount);
+ }
referenceCount--;
- refCount = referenceCount;
-
- if (referenceCount < 0)
- logger.warn("Unmatched Global lock " + name + " reference usage detected, check your code!");
- if (referenceCount == 0)
+ if (referenceCount < 0) {
+ logger.warn("Unmatched internal lock {} reference usage detected (reference count: {}, " +
+ "lock count: {}), check your code!", name, referenceCount, lockCount);
+ } else if (referenceCount < 1) {
needToRemove = true;
+ }
}
- if (needToRemove)
+ if (needToRemove) {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Need to release internal lock {}", name);
+ }
releaseInternLock(name);
+ }
+ if (logger.isDebugEnabled()) {
+ logger.debug("Released reference for lock {}, reference count: {}", name, referenceCount);
+ }
+ return referenceCount;
+ }
- return refCount;
+ public static boolean isLockAvailable(String name) {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Checking lock availability for {}", name);
+ }
+ boolean result = false;
+ try {
+ result = DbUtil.isFreeLock(name);
+ } finally {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Result of checking lock availability for {}: {}", name, result);
+ }
+ }
+ return result;
}
+ /**
+ * Registers internal lock (in memory) object. Does not create any lock in DB yet.
+ *
+ * @param name lock name
+ * @return lock object
+ */
public static GlobalLock getInternLock(String name) {
synchronized (s_lockMap) {
+ GlobalLock lock;
if (s_lockMap.containsKey(name)) {
- GlobalLock lock = s_lockMap.get(name);
- lock.addRef();
- return lock;
+ lock = s_lockMap.get(name);
+ if (logger.isDebugEnabled()) {
+ logger.debug("Internal lock {} already exists with reference count {} and lock count {}",
+ name, lock.referenceCount, lock.lockCount);
+ }
} else {
- GlobalLock lock = new GlobalLock(name);
- lock.addRef();
+ lock = new GlobalLock(name);
+ if (logger.isDebugEnabled()) {
+ logger.debug("Internal lock {} does not exist, adding", name);
+ }
s_lockMap.put(name, lock);
- return lock;
}
+ lock.addRef();
+ if (logger.isDebugEnabled()) {
+ logger.debug("Added reference to internal lock {}, reference count {}, lock count {}",
+ name, lock.referenceCount, lock.lockCount);
+ }
+ return lock;
}
}
+ /**
+ * Unregister internal lock (in memory) object. Does not remove any lock from DB.
+ *
+ * @param name lock name
+ */
private void releaseInternLock(String name) {
synchronized (s_lockMap) {
GlobalLock lock = s_lockMap.get(name);
if (lock != null) {
- if (lock.referenceCount == 0)
+ if (lock.referenceCount == 0) {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Released internal lock {}", name);
+ }
s_lockMap.remove(name);
+ } else {
+ if (logger.isDebugEnabled()) {
+ logger.debug("Not releasing internal lock {} as it has references count: {}, lock count: {}",
+ name, lock.referenceCount, lock.lockCount);
+ }
+ }
} else {
- logger.warn("Releasing " + name + ", but it is already released.");
+ logger.warn("Internal lock {} already released", name);
}
}
}
+ /**
+ * Acquire or join existing DB lock.
+ *
+ * @param timeoutSeconds time in seconds during which lock needs to be obtained (it is not the lock duration)
+ * @return true if lock successfully obtained
+ */
public boolean lock(int timeoutSeconds) {
int remainingMilliSeconds = timeoutSeconds * 1000;
Profiler profiler = new Profiler();
boolean interrupted = false;
try {
while (true) {
synchronized (this) {
- if (ownerThread != null && ownerThread == Thread.currentThread()) {
- logger.warn("Global lock re-entrance detected");
-
+ if (ownerThread == Thread.currentThread()) {
+ logger.warn("Global lock {} re-entrance detected, owner thread: {}, reference count: {}, " +
+ "lock count: {}", name, getThreadName(ownerThread), referenceCount, lockCount);
+ // if it is re-entrance, then we may have more lock counts than needed?
lockCount++;
- if (logger.isTraceEnabled())
- logger.trace("lock " + name + " is acquired, lock count :" + lockCount);
+ if (logger.isDebugEnabled()) {
+ logger.debug("Global lock {} joined, reference count: {}, lock count: {}",
+ name, referenceCount, lockCount);
+ }
return true;
- }
-
- if (ownerThread != null) {
+ } else if (ownerThread != null) {
profiler.start();
try {
- wait((timeoutSeconds) * 1000L);
+ logger.debug("Waiting {} seconds to acquire global lock {}", timeoutSeconds, name);
+ wait(timeoutSeconds * 1000L);
} catch (InterruptedException e) {
interrupted = true;
}
profiler.stop();
remainingMilliSeconds -= profiler.getDurationInMillis();
- if (remainingMilliSeconds < 0)
+ if (remainingMilliSeconds < 0) {
+ logger.warn("Timeout of {} seconds to acquire global lock {} has been reached, " +
+ "owner thread {}, reference count: {}, lock count: {}", timeoutSeconds, name, getThreadName(ownerThread), referenceCount, lockCount);
return false;
+ }
continue;
} else {
// take ownership temporarily to prevent others enter into stage of acquiring DB lock
ownerThread = Thread.currentThread();
+ // XXX: do we need it here (???)
Review Comment:
Nit: leftover casual "XXX: do we need it here (???)" comment carried over in the moved code.
##########
agent/src/main/java/com/cloud/agent/ServerAttache.java:
##########
@@ -0,0 +1,489 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent;
+
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.Command;
+import com.cloud.agent.transport.Request;
+import com.cloud.agent.transport.Response;
+import com.cloud.exception.AgentUnavailableException;
+import com.cloud.exception.CloudException;
+import com.cloud.exception.OperationTimedoutException;
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.cloud.utils.nio.Link;
+import org.apache.cloudstack.managed.context.ManagedContextRunnable;
+import org.apache.cloudstack.utils.reflectiontostringbuilderutils.ReflectionToStringBuilderUtils;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.ThreadContext;
+
+import java.io.IOException;
+import java.nio.channels.ClosedChannelException;
+import java.security.SecureRandom;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * ServerAttache provides basic server communication commands to be implemented.
+ *
+ * @author mprokopchuk
+ */
+public class ServerAttache {
+ private static final Logger logger = LogManager.getLogger(ServerAttache.class);
+
+ private static final ScheduledExecutorService s_listenerExecutor = Executors.newScheduledThreadPool(10,
+ new NamedThreadFactory("ListenerTimer"));
+
+ protected static Comparator<Request> s_reqComparator = (o1, o2) -> {
+ long seq1 = o1.getSequence();
+ long seq2 = o2.getSequence();
+ if (seq1 < seq2) {
+ return -1;
+ } else if (seq1 > seq2) {
+ return 1;
+ } else {
+ return 0;
+ }
+ };
+
+ protected static Comparator<Object> s_seqComparator = (o1, o2) -> {
+ long seq1 = ((Request) o1).getSequence();
+ long seq2 = (Long) o2;
+ if (seq1 < seq2) {
+ return -1;
+ } else if (seq1 > seq2) {
+ return 1;
+ } else {
+ return 0;
+ }
+ };
+
+ private static final Random s_rand = new SecureRandom();
+ protected String _name;
+ private Link _link;
+ protected ConcurrentHashMap<Long, ServerListener> _waitForList;
+ protected ConcurrentHashMap<Long, java.util.concurrent.ScheduledFuture<?>> _alarmFutures;
+ protected LinkedList<Request> _requests;
+ protected Long _currentSequence;
+ protected long _nextSequence;
+
+ protected ServerAttache(Link link) {
+ _name = link.getIpAddress();
+ _link = link;
+ _waitForList = new ConcurrentHashMap<>();
+ _alarmFutures = new ConcurrentHashMap<>();
+ _requests = new LinkedList<>();
+ _nextSequence = Long.valueOf(s_rand.nextInt(Short.MAX_VALUE)) << 48;
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ServerAttache %s", ReflectionToStringBuilderUtils.reflectOnlySelectedFields(this,
+ "_name"));
+ }
+
+ public synchronized long getNextSequence() {
+ return ++_nextSequence;
+ }
+
+ protected synchronized void addRequest(Request req) {
+ int index = findRequest(req);
+ assert (index < 0) : "How can we get index again? " + index + ":" + req.toString();
+ _requests.add(-index - 1, req);
+ }
+
+ protected void cancel(Request req) {
+ cancel(req.getSequence());
+ }
+
+ protected synchronized void cancel(long seq) {
+ logger.debug(log(seq, "Cancelling."));
+
+ ServerListener listener = _waitForList.remove(seq);
+ if (listener != null) {
+ listener.processDisconnect();
+ }
+ int index = findRequest(seq);
+ if (index >= 0) {
+ _requests.remove(index);
+ }
+ }
+
+ protected synchronized int findRequest(Request req) {
+ return Collections.binarySearch(_requests, req, s_reqComparator);
+ }
+
+ protected synchronized int findRequest(long seq) {
+ return Collections.binarySearch(_requests, seq, s_seqComparator);
+ }
+
+ protected String log(long seq, String msg) {
+ return "Seq " + _name + "-" + seq + ": " + msg;
+ }
+
+ protected void registerListener(long seq, ServerListener listener) {
+ if (logger.isTraceEnabled()) {
+ logger.trace(log(seq, "Registering listener"));
+ }
+ if (listener.getTimeout() != -1) {
+ java.util.concurrent.ScheduledFuture<?> alarmFuture =
+ s_listenerExecutor.schedule(new Alarm(seq), listener.getTimeout(), TimeUnit.SECONDS);
+ _alarmFutures.put(seq, alarmFuture);
+ }
+ _waitForList.put(seq, listener);
+ }
+
+ protected ServerListener unregisterListener(long sequence) {
+ if (logger.isTraceEnabled()) {
+ logger.trace(log(sequence, "Unregistering listener"));
+ }
+ java.util.concurrent.ScheduledFuture<?> alarmFuture = _alarmFutures.remove(sequence);
+ if (alarmFuture != null) {
+ alarmFuture.cancel(false);
+ }
+ return _waitForList.remove(sequence);
+ }
+
+ protected ServerListener getListener(long sequence) {
+ return _waitForList.get(sequence);
+ }
+
+ public String getName() {
+ return _name;
+ }
+
+ public int getQueueSize() {
+ return _requests.size();
+ }
+
+ public boolean processAnswers(long seq, Response resp) {
+ resp.logD("Processing: ", true);
+ boolean processed = false;
+ Answer[] answers = resp.getAnswers();
+ try {
+ ServerListener monitor = getListener(seq);
+ if (monitor == null) {
+ if (answers[0] != null && answers[0].getResult()) {
+ processed = true;
+ }
+ logger.debug(log(seq, "Unable to find listener."));
+ } else {
+ processed = monitor.processAnswers(seq, answers);
+ logger.trace(log(seq, (processed ? "" : " did not ") + " processed "));
+ if (!monitor.isRecurring()) {
+ unregisterListener(seq);
+ }
+ }
+ } finally {
+ // we should always trigger next command execution, even in failure cases - otherwise in exception case
+ // all the remaining will be stuck in the sync queue forever
+ if (resp.executeInSequence()) {
+ sendNext(seq);
+ }
+ }
+ return processed;
+ }
+
+ protected void cancelAllCommands() {
+ for (Iterator<Map.Entry<Long, ServerListener>> it = _waitForList.entrySet().iterator(); it.hasNext(); ) {
+ Map.Entry<Long, ServerListener> entry = it.next();
+ it.remove();
+
+ long seq = entry.getKey();
+ java.util.concurrent.ScheduledFuture<?> alarmFuture = _alarmFutures.remove(seq);
+ if (alarmFuture != null) {
+ alarmFuture.cancel(false);
+ }
+ ServerListener monitor = entry.getValue();
+ logger.debug(log(seq, "Sending disconnect to " + monitor.getClass()));
+ monitor.processDisconnect();
+ }
+ }
+
+ public void cleanup() {
+ cancelAllCommands();
+ _requests.clear();
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ // Return false straight away.
+ if (obj == null) {
+ return false;
+ }
+ // No need to handle a ClassCastException. If the classes are different, then equals can return false
+ // straight ahead.
+ if (this.getClass() != obj.getClass()) {
+ return false;
+ }
+ ServerAttache that = (ServerAttache) obj;
+ return _name.equals(that.getName());
+ }
+
+ public void send(Request req, ServerListener listener) throws CloudException {
+ long seq = req.getSequence();
+ if (listener != null) {
+ registerListener(seq, listener);
+ }
+
+ synchronized (this) {
+ try {
+ if (isClosed()) {
+ throw new CloudException("The link to the server " + _name + " has been closed", new ClosedChannelException());
+ }
+
+ if (req.executeInSequence() && _currentSequence != null) {
+ req.logD("Waiting for Seq " + _currentSequence + " Scheduling: ", true);
+ addRequest(req);
+ return;
+ }
+
+ // If we got to here either we're not supposed to set the _currentSequence or it is null already.
+ req.logD("Sending ", true);
+ send(req);
+
+ if (req.executeInSequence() && _currentSequence == null) {
+ _currentSequence = seq;
+ logger.trace(log(seq, " is current sequence"));
+ }
+ } catch (CloudException e) {
+ logger.info(log(seq, "Unable to send due to " + e.getMessage()), e);
+ cancel(seq);
+ throw e;
+ } catch (Exception e) {
+ logger.warn(log(seq, "Unable to send due to " + e.getMessage()), e);
+ cancel(seq);
+ throw new CloudException("Problem due to other exception " + e.getMessage(), e);
+ }
+ }
+ }
+
+ public Answer[] send(Request req, int wait) throws CloudException {
+ if (req.getSequence() <= 0) {
+ req.setSequence(getNextSequence());
+ }
+ SynchronousListener sl = new SynchronousListener();
+ sl.setTimeout(wait + 5);
+ send(req, sl);
+
+ long seq = req.getSequence();
+ try {
+ for (int i = 0; i < 2; i++) {
+ Answer[] answers = null;
+ try {
+ answers = sl.waitFor(wait);
+ } catch (InterruptedException e) {
+ logger.debug(log(seq, "Interrupted"));
+ }
+ if (answers != null) {
+ new Response(req, answers).logD("Received: ", false);
+ return answers;
+ }
+
+ answers = sl.getAnswers(); // Try it again.
+ if (answers != null) {
+ new Response(req, answers).logD("Received after timeout: ", true);
+ return answers;
+ }
+
+ Long current = _currentSequence;
+ if (current != null && seq != current) {
+ if (logger.isDebugEnabled()) {
+ logger.debug(log(seq, "Waited too long."));
+ }
+ throw new OperationTimedoutException(req.getCommands(), -1, seq, wait, false);
+ }
+ logger.debug(log(seq, "Waiting some more time because this is the current command"));
+ }
+ throw new OperationTimedoutException(req.getCommands(), -1, seq, wait * 2, true);
+ } catch (OperationTimedoutException e) {
+ logger.warn(log(seq, "Timed out on " + req));
+ cancel(seq);
+ Long current = _currentSequence;
+ if (req.executeInSequence() && (current != null && current == seq)) {
+ sendNext(seq);
+ }
+ throw e;
+ } catch (Exception e) {
+ logger.warn(log(seq, "Exception while waiting for answer"), e);
+ cancel(seq);
+ Long current = _currentSequence;
+ if (req.executeInSequence() && (current != null && current == seq)) {
+ sendNext(seq);
+ }
+ throw new OperationTimedoutException(req.getCommands(), -1, seq, wait, false);
+ } finally {
+ unregisterListener(seq);
+ }
+ }
+
+ protected synchronized void sendNext(long seq) {
+ _currentSequence = null;
+ if (_requests.isEmpty()) {
+ if (logger.isDebugEnabled()) {
+ logger.debug(log(seq, "No more commands found"));
+ }
+ return;
+ }
+
+ Request req = _requests.pop();
+ if (logger.isDebugEnabled()) {
+ logger.debug(log(req.getSequence(), "Sending now. is current sequence."));
+ }
+ try {
+ send(req);
+ } catch (CloudException e) {
+ logger.debug(log(req.getSequence(), "Unable to send the next sequence"));
+ cancel(req.getSequence());
+ }
+ _currentSequence = req.getSequence();
+ }
+
+ public void process(Answer[] answers) {
+ //do nothing
+ }
+
+ /**
+ * sends the request asynchronously.
+ *
+ * @param req
+ * @throws AgentUnavailableException
+ */
+ public synchronized void send(final Request req) throws CloudException {
+ try {
+ _link.send(req.toBytes());
+ } catch (ClosedChannelException e) {
+ throw new CloudException("Channel is closed", e);
+ }
+ }
+
+ /**
+ * Process disconnect.
+ */
+ public void disconnect() {
+ synchronized (this) {
+ logger.debug("Processing Disconnect.");
+ if (_link != null) {
+ _link.close();
+ _link.terminated();
+ }
+ _link = null;
+ }
+ cancelAllCommands();
+ _requests.clear();
+ }
+
+ /**
+ * Is the agent closed for more commands?
+ *
+ * @return true if unable to reach agent or false if reachable.
+ */
+ protected boolean isClosed() {
+ return _link == null;
+ }
+
+ public Link getLink() {
+ return _link;
+ }
+
+ public <T> T send(Long agentId, Command[] commands, Class<T> answerType, int asyncCommandTimeoutSec) throws IOException {
+ String logId = Optional.ofNullable(ThreadContext.get("logcontextid"))
+ .filter(String.class::isInstance)
+ .map(String.class::cast)
+ .orElse(null);
+ if (logId != null) {
+ for (Command command : commands) {
+ if (command.getContextParam("logid") == null) {
+ command.setContextParam("logid", logId);
+ }
+ }
+ }
+ Link link = getLink();
+ String commandName = commands[0].getClass().getSimpleName();
+
+ Long id = Optional.ofNullable(agentId).orElse(-1L);
+ Request request = new Request(id, -1, commands, true, false);
+ Answer[] answers;
+ try {
+ answers = send(request, asyncCommandTimeoutSec);
+ } catch (OperationTimedoutException e) {
+ String msg = String.format("Failed to retrieve answer for command %s to %s", commandName, link);
+ logger.error(msg, e);
+ throw new RuntimeException(msg, e);
+ } catch (CloudException e) {
+ String msg = String.format("Failed to send command %s to %s", commandName, link);
+ logger.error(msg, e);
+ Throwable cause = e.getCause();
+ if (cause instanceof ClosedChannelException) {
+ throw (ClosedChannelException) cause;
+ }
+ throw new RuntimeException(msg, e);
+ }
+ if (ArrayUtils.isEmpty(answers)) {
+ String msg = String.format("Received empty %s response from %s", commandName, link);
+ logger.warn(msg);
+ throw new IllegalArgumentException(msg);
+ }
+ Answer answer = answers[0];
+ String details = answer.getDetails();
+ if (!answer.getResult()) {
+ String msg = String.format("Received unsuccessful %s response status from %s: %s", commandName, link,
+ details);
+ logger.warn(msg);
+ throw new IllegalArgumentException(msg);
+ }
+ String answerName = answer.getClass().getSimpleName();
+ if (!answerType.isInstance(answer)) {
+ String msg = String.format("Received unexpected %s response type %s from %s: %s", commandName,
+ answerName, link, details);
+ logger.warn(msg);
+ throw new IllegalArgumentException(msg);
+ }
+ return answerType.cast(answer);
+ }
+
+ protected class Alarm extends ManagedContextRunnable {
+ long _seq;
+
+ public Alarm(long seq) {
+ _seq = seq;
+ }
+
+ @Override
+ protected void runInContext() {
+ try {
+ ServerListener listener = unregisterListener(_seq);
+ if (listener != null) {
+ cancel(_seq);
+ listener.processTimeout(_seq);
+ }
+ } catch (Exception e) {
+ ServerAttache.logger.warn("Exception ", e);
Review Comment:
Nit: vague log message ("Exception " with a trailing space), no context on what actually failed.
##########
engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java:
##########
@@ -762,40 +985,124 @@ public void notifyMonitorsOfNewlyAddedHost(long hostId) {
protected AgentAttache notifyMonitorsOfConnection(final AgentAttache attache, final StartupCommand[] cmds, final boolean forRebalance) throws ConnectionException {
final long hostId = attache.getId();
final HostVO host = _hostDao.findById(hostId);
- for (final Pair<Integer, Listener> monitor : _hostMonitors) {
- logger.debug("Sending Connect to listener: {}, for rebalance: {}", monitor.second().getClass().getSimpleName(), forRebalance);
- for (StartupCommand cmd : cmds) {
- try {
- logger.debug("process connection to issue: {} for host: {}, forRebalance: {}", ReflectionToStringBuilderUtils.reflectOnlySelectedFields(cmd, "id", "type", "msHostList", "connectionTransferred"), hostId, forRebalance);
- monitor.second().processConnect(host, cmd, forRebalance);
- } catch (final ConnectionException ce) {
- if (ce.isSetupError()) {
- logger.warn("Monitor {} says there is an error in the connect process for {} due to {}", monitor.second().getClass().getSimpleName(), hostId, ce.getMessage());
- handleDisconnectWithoutInvestigation(attache, Event.AgentDisconnected, true, true);
- throw ce;
- } else {
- logger.info("Monitor {} says not to continue the connect process for {} due to {}", monitor.second().getClass().getSimpleName(), hostId, ce.getMessage());
- handleDisconnectWithoutInvestigation(attache, Event.ShutdownRequested, true, true);
- return attache;
- }
- } catch (final HypervisorVersionChangedException hvce) {
- handleDisconnectWithoutInvestigation(attache, Event.ShutdownRequested, true, true);
- throw new CloudRuntimeException("Unable to connect " + attache.getId(), hvce);
- } catch (final Exception e) {
- logger.error("Monitor {} says there is an error in the connect process for {} due to {}", monitor.second().getClass().getSimpleName(), hostId, e.getMessage(), e);
- handleDisconnectWithoutInvestigation(attache, Event.AgentDisconnected, true, true);
- throw new CloudRuntimeException("Unable to connect " + attache.getId(), e);
- }
+ Optional<HostVO> hostOpt = Optional.ofNullable(host);
+ String hostName = hostOpt.map(HostVO::getName).orElse(null);
+ String hostUuid = hostOpt.map(HostVO::getUuid).orElse(null);
+ boolean processSuccessful;
+ Profiler processProfiler = new Profiler();
+ processProfiler.start();
+ logger.debug("Process connect for host: {} ({}) for rebalance: {}", hostName, hostUuid, forRebalance);
+ try {
+ processSuccessful = performProcessConnect(attache, cmds, forRebalance, hostName, hostUuid, host);
+ } finally {
+ processProfiler.stop();
+ long processDurationSec = processProfiler.getDurationInMillis() / 1000;
+ logger.debug("Finished process connect for host: {} ({}) for rebalance: {} duration, s: {}",
+ hostName, hostUuid, forRebalance, processDurationSec);
+ }
+ if (processSuccessful) {
+ sendReadyCommand(attache, host, hostId);
+
+ agentStatusTransitTo(host, Event.Ready, _nodeId);
+ attache.ready();
+ }
+ return attache;
+ }
+
+ /**
+ * @return true if commands successfully completed
+ */
+ private boolean performProcessConnect(AgentAttache attache, StartupCommand[] cmds, boolean forRebalance, String hostName, String hostUuid, HostVO host) throws ConnectionException {
+ int monitorsSize = _hostMonitors.size();
+ boolean processSuccessful = true;
+ for (int monitorIndex = 0; monitorIndex < monitorsSize && processSuccessful; monitorIndex++) {
+ Pair<Integer, Listener> monitor = _hostMonitors.get(monitorIndex);
+ Listener listener = monitor.second();
+ String monitorClassName = listener.getClass().getSimpleName();
+ Profiler monitorProfiler = new Profiler();
+ logger.debug("Process connect for host: {} ({}) listener {} ({} of {}) for rebalance: {}",
+ hostName, hostUuid, monitorClassName, monitorIndex, monitorsSize, forRebalance);
+ try {
+ monitorProfiler.start();
+ processSuccessful = performProcessConnect(attache, cmds, forRebalance, hostName, hostUuid, monitorClassName, listener, host);
+ } finally {
+ monitorProfiler.stop();
+ long monitorDurationSec = monitorProfiler.getDurationInMillis() / 1000;
+ logger.debug("Finished process connect for host: {} ({}) listener: {} ({} of {}) for rebalance: {} duration, s: {}",
+ hostName, hostUuid, monitorClassName, monitorIndex, monitorsSize, forRebalance, monitorDurationSec);
+ }
+ }
+ return processSuccessful;
+ }
+
+ /**
+ * @return true if commands successfully completed
+ */
+ private boolean performProcessConnect(AgentAttache attache, StartupCommand[] cmds, boolean forRebalance, String hostName, String hostUuid, String monitorClassName, Listener listener, HostVO host) throws ConnectionException {
+ boolean processSuccessful = true;
+ int commandsSize = cmds.length;
+ for (int commandIndex = 0; commandIndex < commandsSize && processSuccessful; commandIndex++) {
+ StartupCommand command = cmds[commandIndex];
+ String commandClassName = command.getClass().getSimpleName();
+ boolean connTransferred = command.isConnectionTransferred();
+ long processStart = System.currentTimeMillis();
+ try {
+ logger.debug("Process command: {} ({} of {}) in connect for host: {} ({}) listener: {} for rebalance: {} connection transferred: {}",
+ commandClassName, commandIndex, commandsSize, hostName, hostUuid, monitorClassName, forRebalance, connTransferred);
+ listener.processConnect(host, command, forRebalance);
+ } catch (Exception e) {
+ handleConnectionException(attache, e, processStart, host, monitorClassName);
+ processSuccessful = false;
+ } finally {
+ long commandDurationSec = (System.currentTimeMillis() - processStart) / 1000;
+ logger.debug("Finished processing command: {} ({} of {}) in connect for host: {} ({}) listener: {} for rebalance: {} connection transferred: {} duration, s: {}",
+ commandClassName, commandIndex, commandsSize, hostName, hostUuid, monitorClassName, forRebalance, connTransferred, commandDurationSec);
+ }
+ }
+ return processSuccessful;
+ }
+
+ private void handleConnectionException(AgentAttache attache, Exception e, long processStart, HostVO host, String monitorClassName) throws ConnectionException {
+ long processDurationMs = System.currentTimeMillis() - processStart;
+ StringBuilder summaryBuilder = getSummaryMsgBuilder("Failed to connect Host",
+ EventTypes.EVENT_HOST_RECONNECT, host.getUuid(), host.getName(), null,
+ e.getMessage(), monitorClassName, processDurationMs);
+ logger.fatal(summaryBuilder.toString(), e);
+
+ // add alert to the Host here
+ if (e instanceof ConnectionException) {
+ ConnectionException ce = (ConnectionException) e;
+ // XXX: in case of Storage Pool issue we are ending up here
Review Comment:
Nit: leftover `XXX` comment carried over in the moved code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-4619916053
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18146
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "sureshanaparti (via GitHub)" <gi...@apache.org>.
sureshanaparti commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-4620383653
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "nvazquez (via GitHub)" <gi...@apache.org>.
nvazquez commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-5724382510
Hi @sureshanaparti can you please fix the merge conflicts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#discussion_r4046648818
##########
agent/src/main/java/com/cloud/agent/AgentShell.java:
##########
@@ -424,11 +434,15 @@ public void init(String[] args) throws ConfigurationException {
_properties.put(cmdLineProp.getKey(), cmdLineProp.getValue());
}
- LOGGER.info("Defaulting to the constant time backoff algorithm");
- _backoff = new ConstantTimeBackoff();
- Map<String, Object> map = new HashMap<>();
- map.put("seconds", _properties.getProperty("backoff.seconds"));
- _backoff.configure("ConstantTimeBackoff", map);
+ try {
+ LOGGER.info("Creating backoff delay algorithm implementation");
+ setBackoffAlgorithm(BackoffFactory.create(_properties));
+ LOGGER.info("Created {} delay algorithm implementation", getBackoffAlgorithm().getClass().getName());
+ } catch (RuntimeException e) {
+ String msg = String.format("Failed to create backoff with provided properties %s, failing back to default", _properties);
Review Comment:
The fallback message formats the entire properties object, which can include the persisted keystore passphrase. This leaks the same credential on the WARN path when backoff configuration is invalid; do not include `_properties` in the message.
##########
engine/orchestration/src/test/java/com/cloud/agent/manager/AgentManagerImplTest.java:
##########
@@ -52,11 +73,12 @@ public class AgentManagerImplTest {
@Before
public void setUp() throws Exception {
- host = Mockito.spy(new HostVO("some-Uuid"));
- Mockito.when(host.getId()).thenReturn(1L);
+ host = new HostVO("some-Uuid");
+ Mockito.when(host.getId()).thenReturn(1L)
+ FieldUtils.writeField(host, "id", HOST_ID, true);
Review Comment:
`host` is now a real `HostVO`, not a Mockito mock or spy, so this `when(...)` call throws `MissingMethodInvocationException`; as written it also lacks a semicolon and prevents compilation. The following reflection write already sets the ID, so remove the stubbing line.
##########
utils/src/main/java/com/cloud/utils/nio/HandlerFactory.java:
##########
@@ -32,6 +32,6 @@ default int getMaxConcurrentNewConnectionsCount() {
default int getNewConnectionsCount() {
return 0;
}
- default void registerNewConnection(SocketAddress address) {}
- default void unregisterNewConnection(SocketAddress address) {}
+ default void registerNewConnection(InetSocketAddress address) {}
+ default void unregisterNewConnection(InetSocketAddress address) {}
Review Comment:
`HandlerFactory` is a public interface and previously accepted the general `SocketAddress`; narrowing these default-method parameters to `InetSocketAddress` breaks existing implementations that override the old signatures (their methods will no longer override and the new defaults will run). Keep the public parameter type as `SocketAddress` and narrow only at the implementation boundary.
##########
framework/cluster/src/main/java/com/cloud/cluster/dao/ManagementServerHostDaoImpl.java:
##########
@@ -64,6 +65,11 @@ public void invalidateRunSession(long id, long runid) {
}
}
+ @Override
+ public List<ManagementServerHostVO> findAllIncludingRemoved() {
+ return listIncludingRemovedBy(ActiveSearch.create());
Review Comment:
`ActiveSearch` filters `removed IS NULL` and requires a recent `lastUpdateTime`, so this method does not return “all management servers (including down and removed)” as its new contract says. It excludes removed and stale/down rows; use the inherited `listAllIncludingRemoved()` instead.
##########
plugins/maintenance/src/main/java/org/apache/cloudstack/maintenance/ManagementServerMaintenanceManagerImpl.java:
##########
@@ -466,9 +466,20 @@ public ManagementServerMaintenanceResponse prepareForMaintenance(PrepareForMaint
boolean ignoreMaintenanceHosts = ManagementServerMaintenanceIgnoreMaintenanceHosts.value();
if (indirectAgentLB.haveAgentBasedHosts(msHost.getMsid(), ignoreMaintenanceHosts)) {
List<String> indirectAgentMsList = indirectAgentLB.getManagementServerList();
- indirectAgentMsList.remove(msHost.getServiceIP());
- List<String> nonUpMsList = msHostDao.listNonUpStateMsIPs();
- indirectAgentMsList.removeAll(nonUpMsList);
+ // Remove current server by both hostname and IP since the list could contain either
+ if (msHost.getName() != null) {
+ indirectAgentMsList.remove(msHost.getName());
+ }
+ if (msHost.getServiceIP() != null) {
+ indirectAgentMsList.remove(msHost.getServiceIP());
+ }
Review Comment:
`indirectAgentMsList` contains configured addresses, but this removes the persisted canonical hostname/IP. For hostname aliases neither removal is guaranteed to match, so maintenance can send agents a list that still contains the server being put into maintenance. Resolve the current node to its configured address before removing it.
##########
framework/cluster/src/main/java/com/cloud/cluster/ClusterServiceServletHttpHandler.java:
##########
@@ -113,6 +117,14 @@ private void writeResponse(HttpResponse response, int statusCode, String content
response.setEntity(body);
}
+ private void logRequest(HttpRequest request, String requestBody) {
+ Optional<HttpRequest> requestOpt = Optional.ofNullable(request);
+ Optional<RequestLine> requestLineOpt = requestOpt.map(HttpRequest::getRequestLine);
+ String method = requestLineOpt.map(RequestLine::getMethod).orElse(null);
+ String uri = requestLineOpt.map(RequestLine::getUri).orElse(null);
+ logger.debug("{} {} {}", method, uri, requestBody);
Review Comment:
This logs the complete cluster POST body, including the `gsonPackage` command/answer JSON. Commands on this channel can contain credential fields, so the new DEBUG statement creates a plaintext secret sink in management-server logs; log request metadata or redact the payload.
##########
agent/src/main/java/com/cloud/agent/ServerAttache.java:
##########
@@ -0,0 +1,493 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent;
+
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.Command;
+import com.cloud.agent.transport.Request;
+import com.cloud.agent.transport.Response;
+import com.cloud.exception.AgentUnavailableException;
+import com.cloud.exception.CloudException;
+import com.cloud.exception.OperationTimedoutException;
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.cloud.utils.nio.Link;
+import org.apache.cloudstack.managed.context.ManagedContextRunnable;
+import org.apache.cloudstack.threadcontext.ThreadContextCommandUtil;
+import org.apache.cloudstack.utils.reflectiontostringbuilderutils.ReflectionToStringBuilderUtils;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.ThreadContext;
+
+import java.io.IOException;
+import java.nio.channels.ClosedChannelException;
+import java.security.SecureRandom;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * ServerAttache provides basic server communication commands to be implemented.
+ *
+ * @author mprokopchuk
+ */
+public class ServerAttache {
+ private static final Logger logger = LogManager.getLogger(ServerAttache.class);
+
+ private static final ScheduledExecutorService s_listenerExecutor = Executors.newScheduledThreadPool(10,
+ new NamedThreadFactory("ListenerTimer"));
+
+ protected static Comparator<Request> s_reqComparator = (o1, o2) -> {
+ long seq1 = o1.getSequence();
+ long seq2 = o2.getSequence();
+ if (seq1 < seq2) {
+ return -1;
+ } else if (seq1 > seq2) {
+ return 1;
+ } else {
+ return 0;
+ }
+ };
+
+ protected static Comparator<Object> s_seqComparator = (o1, o2) -> {
+ long seq1 = ((Request) o1).getSequence();
+ long seq2 = (Long) o2;
+ if (seq1 < seq2) {
+ return -1;
+ } else if (seq1 > seq2) {
+ return 1;
+ } else {
+ return 0;
+ }
+ };
+
+ private static final Random s_rand = new SecureRandom();
+ protected String _name;
+ private Link _link;
+ protected ConcurrentHashMap<Long, ServerListener> _waitForList;
+ protected ConcurrentHashMap<Long, java.util.concurrent.ScheduledFuture<?>> _alarmFutures;
+ protected LinkedList<Request> _requests;
+ protected Long _currentSequence;
+ protected long _nextSequence;
+
+ protected ServerAttache(Link link) {
+ _name = link.getIpAddress();
+ _link = link;
+ _waitForList = new ConcurrentHashMap<>();
+ _alarmFutures = new ConcurrentHashMap<>();
+ _requests = new LinkedList<>();
+ _nextSequence = Long.valueOf(s_rand.nextInt(Short.MAX_VALUE)) << 48;
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ServerAttache %s", ReflectionToStringBuilderUtils.reflectOnlySelectedFields(this,
+ "_name"));
+ }
+
+ public synchronized long getNextSequence() {
+ return ++_nextSequence;
+ }
+
+ protected synchronized void addRequest(Request req) {
+ int index = findRequest(req);
+ assert (index < 0) : "How can we get index again? " + index + ":" + req.toString();
+ _requests.add(-index - 1, req);
+ }
+
+ protected void cancel(Request req) {
+ cancel(req.getSequence());
+ }
+
+ protected synchronized void cancel(long seq) {
+ logger.debug(log(seq, "Cancelling."));
+
+ ServerListener listener = _waitForList.remove(seq);
+ if (listener != null) {
+ listener.processDisconnect();
+ }
Review Comment:
On an asynchronous send failure, `cancel(seq)` removes the listener but leaves its scheduled alarm in `_alarmFutures`; unlike `unregisterListener`, it does not cancel the future. Repeated connection failures can retain one delayed task per request until the timeout, so cancel the corresponding future here as well.
##########
core/src/main/java/org/apache/cloudstack/threadcontext/ThreadContextCommandUtil.java:
##########
@@ -0,0 +1,63 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.threadcontext;
+
+import com.cloud.agent.api.Command;
+import com.cloud.utils.StringUtils;
+import org.apache.logging.log4j.ThreadContext;
+
+/**
+ * Utility class for Command-specific MDC operations.
+ * This class handles propagation of MDC values to and from Command objects.
+ *
+ * @author mprokopchuk
+ */
+public class ThreadContextCommandUtil {
+
+ /**
+ * Propagate UUID and log context ID from Command trace context to MDC.
+ *
+ * @param cmd the command containing trace context parameters
+ */
+ public static void propagateContextFromCommand(Command cmd) {
+ if (cmd != null) {
+ ThreadContextUtil.setLogContextId(cmd.getTraceContextParam(ThreadContextUtil.CONTEXT_LOG_ID_KEY));
+ ThreadContextUtil.setUuid(cmd.getTraceContextParam(ThreadContextUtil.CONTEXT_UUID_KEY));
Review Comment:
These helpers only write non-empty values, and agent worker threads do not clear MDC between requests. A command without trace context can therefore inherit the UUID or log context from the previous command, misattributing logs and any subsequently propagated command; clear each MDC key when its command parameter is absent.
##########
server/src/main/java/org/apache/cloudstack/agent/lb/IndirectAgentLBServiceImpl.java:
##########
@@ -331,7 +332,9 @@ public void propagateMSListToAgents(boolean triggerHostLB) {
zoneHostIds.addAll(hostIds);
}
zoneHostIds.sort(Comparator.comparingLong(x -> x));
- final List<String> avoidMsList = mshostDao.listNonUpStateMsIPs();
+
+ final List<String> avoidMsList = agentManager.getAvoidMsList();
Review Comment:
Hostname mode sends the configured address list to agents, but this avoid list is built from `mshost.name`, which is the canonical hostname persisted by ClusterManager. With a configured CNAME/alias that differs from that canonical name, the down server's name is not present in the agent's list, so it is not avoided during rebalancing. Build the avoid list in the same configured-address namespace.
##########
core/src/main/java/com/cloud/agent/api/AgentConnectStatusAnswer.java:
##########
@@ -0,0 +1,65 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+package com.cloud.agent.api;
+
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.utils.db.GlobalLock;
+
+/**
+ * Answer for {@link AgentConnectStatusCommand}.
+ *
+ * @author mprokopchuk
+ */
+public class AgentConnectStatusAnswer extends Answer {
+
+ /**
+ * {@link Boolean#TRUE} means host has {@link GlobalLock#lock(int)} acquired, otherwise {@link Boolean#FALSE},
+ * and null if there is an error during executing {@link AgentConnectStatusCommand}.
Review Comment:
This documentation reverses the value returned by the implementation: `AgentManagerImpl` sets it from `GlobalLock.isLockAvailable()`, which is true when `IS_FREE_LOCK` reports that no server holds the lock. `HostStatusTask` also treats true as permission to send startup. Document true as “lock available,” not “lock acquired,” to avoid unsafe changes based on this contract.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#discussion_r3913181040
##########
agent/src/main/java/com/cloud/agent/AgentShell.java:
##########
@@ -410,7 +420,7 @@ public void init(String[] args) throws ConfigurationException {
if (LOGGER.isDebugEnabled()) {
List<String> properties = Collections.list((Enumeration<String>)_properties.propertyNames());
for (String property : properties) {
- LOGGER.debug("Found property: {}", property);
+ LOGGER.debug("Found property: {}, value: {}", property, _properties.getProperty(property));
}
Review Comment:
This debug logging prints *all* agent properties including their values. Agent properties commonly include sensitive data (passwords, keys, tokens), so emitting them to logs is a security risk and can leak secrets in support bundles.
Consider logging only the property names (previous behavior), or explicitly redacting known-sensitive keys before logging values.
##########
framework/cluster/src/main/java/com/cloud/cluster/ClusterServiceServletHttpHandler.java:
##########
@@ -113,6 +117,14 @@ private void writeResponse(HttpResponse response, int statusCode, String content
response.setEntity(body);
}
+ private void logRequest(HttpRequest request, String requestBody) {
+ Optional<HttpRequest> requestOpt = Optional.ofNullable(request);
+ Optional<RequestLine> requestLineOpt = requestOpt.map(HttpRequest::getRequestLine);
+ String method = requestLineOpt.map(RequestLine::getMethod).orElse(null);
+ String uri = requestLineOpt.map(RequestLine::getUri).orElse(null);
+ logger.debug("{} {} {}", method, uri, requestBody);
+ }
Review Comment:
`logRequest` logs the full raw POST body at DEBUG level. Cluster-service requests can include operational identifiers and potentially sensitive parameters; logging the entire body at DEBUG makes accidental data exposure much more likely (and duplicates the per-parameter TRACE logging above).
Suggestion: log only method/URI (or body length / a sanitized subset) unless TRACE is enabled and the data is explicitly safe to log.
##########
utils/src/main/java/com/cloud/utils/backoff/BackoffFactory.java:
##########
@@ -0,0 +1,96 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.utils.backoff;
+
+import com.cloud.utils.backoff.impl.ConstantTimeBackoff;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import javax.naming.ConfigurationException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+import java.util.stream.Collectors;
+
+/**
+ * Backoff implementation factory.
+ *
+ * @author mprokopchuk
+ */
+public interface BackoffFactory {
+ Logger logger = LogManager.getLogger(BackoffFactory.class);
+ /**
+ * Property name for the implementation class (that extends {@link BackoffAlgorithm}) to be used either
+ * by {@code agent.properties} file or by configuration key.
+ */
+ String BACKOFF_IMPLEMENTATION_KEY = "backoff.implementation";
+
+ /**
+ * Default backoff implementation class name ({@link ConstantTimeBackoff}).
+ */
+ String DEFAULT_BACKOFF_IMPLEMENTATION = ConstantTimeBackoff.class.getName();
+
+ /**
+ * Creates default {@link BackoffAlgorithm} implementation object ({@link ConstantTimeBackoff}).
+ *
+ * @param properties configuration properties
+ * @return {@link BackoffAlgorithm} implementation object
+ */
+ static BackoffAlgorithm createDefault(Properties properties) {
+ Properties newProperties = new Properties(properties);
+ newProperties.put(BACKOFF_IMPLEMENTATION_KEY, DEFAULT_BACKOFF_IMPLEMENTATION);
+ return create(newProperties);
+ }
Review Comment:
`createDefault` currently uses `new Properties(properties)`, which sets the passed `properties` as *defaults* rather than copying entries. Since `create(Properties)` builds params from `properties.entrySet()`, the default backoff will be created with only `backoff.implementation` set and will silently drop other backoff-related settings (e.g. `backoff.seconds`, min/max delay), changing behavior when falling back to the default implementation.
Copy the provided properties into a new `Properties` instance before overriding the implementation key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "kiranchavala (via GitHub)" <gi...@apache.org>.
kiranchavala commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-5931931931
cc @nvazquez
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "Damans227 (via GitHub)" <gi...@apache.org>.
Damans227 commented on code in PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#discussion_r4158489147
##########
agent/src/main/java/com/cloud/agent/ServerAttache.java:
##########
@@ -0,0 +1,489 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent;
+
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.Command;
+import com.cloud.agent.transport.Request;
+import com.cloud.agent.transport.Response;
+import com.cloud.exception.AgentUnavailableException;
+import com.cloud.exception.CloudException;
+import com.cloud.exception.OperationTimedoutException;
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.cloud.utils.nio.Link;
+import org.apache.cloudstack.managed.context.ManagedContextRunnable;
+import org.apache.cloudstack.utils.reflectiontostringbuilderutils.ReflectionToStringBuilderUtils;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.ThreadContext;
+
+import java.io.IOException;
+import java.nio.channels.ClosedChannelException;
+import java.security.SecureRandom;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * ServerAttache provides basic server communication commands to be implemented.
+ *
+ * @author mprokopchuk
Review Comment:
can we drop the author tags too? theyre still here and in a few of the other new files
##########
agent/src/main/java/com/cloud/agent/ServerAttache.java:
##########
@@ -0,0 +1,489 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent;
+
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.Command;
+import com.cloud.agent.transport.Request;
+import com.cloud.agent.transport.Response;
+import com.cloud.exception.AgentUnavailableException;
+import com.cloud.exception.CloudException;
+import com.cloud.exception.OperationTimedoutException;
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.cloud.utils.nio.Link;
+import org.apache.cloudstack.managed.context.ManagedContextRunnable;
+import org.apache.cloudstack.utils.reflectiontostringbuilderutils.ReflectionToStringBuilderUtils;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.ThreadContext;
+
+import java.io.IOException;
+import java.nio.channels.ClosedChannelException;
+import java.security.SecureRandom;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * ServerAttache provides basic server communication commands to be implemented.
+ *
+ * @author mprokopchuk
+ */
+public class ServerAttache {
+ private static final Logger logger = LogManager.getLogger(ServerAttache.class);
+
+ private static final ScheduledExecutorService s_listenerExecutor = Executors.newScheduledThreadPool(10,
+ new NamedThreadFactory("ListenerTimer"));
+
+ protected static Comparator<Request> s_reqComparator = (o1, o2) -> {
+ long seq1 = o1.getSequence();
+ long seq2 = o2.getSequence();
+ if (seq1 < seq2) {
+ return -1;
+ } else if (seq1 > seq2) {
+ return 1;
+ } else {
+ return 0;
+ }
+ };
+
+ protected static Comparator<Object> s_seqComparator = (o1, o2) -> {
+ long seq1 = ((Request) o1).getSequence();
+ long seq2 = (Long) o2;
+ if (seq1 < seq2) {
+ return -1;
+ } else if (seq1 > seq2) {
+ return 1;
+ } else {
+ return 0;
+ }
+ };
+
+ private static final Random s_rand = new SecureRandom();
+ protected String _name;
+ private Link _link;
+ protected ConcurrentHashMap<Long, ServerListener> _waitForList;
+ protected ConcurrentHashMap<Long, java.util.concurrent.ScheduledFuture<?>> _alarmFutures;
+ protected LinkedList<Request> _requests;
+ protected Long _currentSequence;
+ protected long _nextSequence;
+
+ protected ServerAttache(Link link) {
+ _name = link.getIpAddress();
+ _link = link;
+ _waitForList = new ConcurrentHashMap<>();
+ _alarmFutures = new ConcurrentHashMap<>();
+ _requests = new LinkedList<>();
+ _nextSequence = Long.valueOf(s_rand.nextInt(Short.MAX_VALUE)) << 48;
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ServerAttache %s", ReflectionToStringBuilderUtils.reflectOnlySelectedFields(this,
+ "_name"));
+ }
+
+ public synchronized long getNextSequence() {
+ return ++_nextSequence;
+ }
+
+ protected synchronized void addRequest(Request req) {
+ int index = findRequest(req);
+ assert (index < 0) : "How can we get index again? " + index + ":" + req.toString();
+ _requests.add(-index - 1, req);
+ }
+
+ protected void cancel(Request req) {
+ cancel(req.getSequence());
+ }
+
+ protected synchronized void cancel(long seq) {
+ logger.debug(log(seq, "Cancelling."));
+
+ ServerListener listener = _waitForList.remove(seq);
+ if (listener != null) {
+ listener.processDisconnect();
+ }
+ int index = findRequest(seq);
+ if (index >= 0) {
+ _requests.remove(index);
+ }
+ }
+
+ protected synchronized int findRequest(Request req) {
+ return Collections.binarySearch(_requests, req, s_reqComparator);
+ }
+
+ protected synchronized int findRequest(long seq) {
+ return Collections.binarySearch(_requests, seq, s_seqComparator);
+ }
+
+ protected String log(long seq, String msg) {
+ return "Seq " + _name + "-" + seq + ": " + msg;
+ }
+
+ protected void registerListener(long seq, ServerListener listener) {
+ if (logger.isTraceEnabled()) {
+ logger.trace(log(seq, "Registering listener"));
+ }
+ if (listener.getTimeout() != -1) {
+ java.util.concurrent.ScheduledFuture<?> alarmFuture =
+ s_listenerExecutor.schedule(new Alarm(seq), listener.getTimeout(), TimeUnit.SECONDS);
+ _alarmFutures.put(seq, alarmFuture);
+ }
+ _waitForList.put(seq, listener);
+ }
+
+ protected ServerListener unregisterListener(long sequence) {
+ if (logger.isTraceEnabled()) {
+ logger.trace(log(sequence, "Unregistering listener"));
+ }
+ java.util.concurrent.ScheduledFuture<?> alarmFuture = _alarmFutures.remove(sequence);
+ if (alarmFuture != null) {
+ alarmFuture.cancel(false);
+ }
+ return _waitForList.remove(sequence);
+ }
+
+ protected ServerListener getListener(long sequence) {
+ return _waitForList.get(sequence);
+ }
+
+ public String getName() {
+ return _name;
+ }
+
+ public int getQueueSize() {
+ return _requests.size();
+ }
+
+ public boolean processAnswers(long seq, Response resp) {
+ resp.logD("Processing: ", true);
+ boolean processed = false;
+ Answer[] answers = resp.getAnswers();
+ try {
+ ServerListener monitor = getListener(seq);
+ if (monitor == null) {
+ if (answers[0] != null && answers[0].getResult()) {
+ processed = true;
+ }
+ logger.debug(log(seq, "Unable to find listener."));
+ } else {
+ processed = monitor.processAnswers(seq, answers);
+ logger.trace(log(seq, (processed ? "" : " did not ") + " processed "));
Review Comment:
ok thats fixed, thanks
##########
agent/src/main/java/com/cloud/agent/ServerAttache.java:
##########
@@ -0,0 +1,489 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.agent;
+
+import com.cloud.agent.api.Answer;
+import com.cloud.agent.api.Command;
+import com.cloud.agent.transport.Request;
+import com.cloud.agent.transport.Response;
+import com.cloud.exception.AgentUnavailableException;
+import com.cloud.exception.CloudException;
+import com.cloud.exception.OperationTimedoutException;
+import com.cloud.utils.concurrency.NamedThreadFactory;
+import com.cloud.utils.nio.Link;
+import org.apache.cloudstack.managed.context.ManagedContextRunnable;
+import org.apache.cloudstack.utils.reflectiontostringbuilderutils.ReflectionToStringBuilderUtils;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.ThreadContext;
+
+import java.io.IOException;
+import java.nio.channels.ClosedChannelException;
+import java.security.SecureRandom;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Random;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+/**
+ * ServerAttache provides basic server communication commands to be implemented.
+ *
+ * @author mprokopchuk
+ */
+public class ServerAttache {
+ private static final Logger logger = LogManager.getLogger(ServerAttache.class);
+
+ private static final ScheduledExecutorService s_listenerExecutor = Executors.newScheduledThreadPool(10,
+ new NamedThreadFactory("ListenerTimer"));
+
+ protected static Comparator<Request> s_reqComparator = (o1, o2) -> {
+ long seq1 = o1.getSequence();
+ long seq2 = o2.getSequence();
+ if (seq1 < seq2) {
+ return -1;
+ } else if (seq1 > seq2) {
+ return 1;
+ } else {
+ return 0;
+ }
+ };
+
+ protected static Comparator<Object> s_seqComparator = (o1, o2) -> {
+ long seq1 = ((Request) o1).getSequence();
+ long seq2 = (Long) o2;
+ if (seq1 < seq2) {
+ return -1;
+ } else if (seq1 > seq2) {
+ return 1;
+ } else {
+ return 0;
+ }
+ };
+
+ private static final Random s_rand = new SecureRandom();
+ protected String _name;
+ private Link _link;
+ protected ConcurrentHashMap<Long, ServerListener> _waitForList;
+ protected ConcurrentHashMap<Long, java.util.concurrent.ScheduledFuture<?>> _alarmFutures;
+ protected LinkedList<Request> _requests;
+ protected Long _currentSequence;
+ protected long _nextSequence;
+
+ protected ServerAttache(Link link) {
+ _name = link.getIpAddress();
+ _link = link;
+ _waitForList = new ConcurrentHashMap<>();
+ _alarmFutures = new ConcurrentHashMap<>();
+ _requests = new LinkedList<>();
+ _nextSequence = Long.valueOf(s_rand.nextInt(Short.MAX_VALUE)) << 48;
+ }
+
+ @Override
+ public String toString() {
+ return String.format("ServerAttache %s", ReflectionToStringBuilderUtils.reflectOnlySelectedFields(this,
+ "_name"));
+ }
+
+ public synchronized long getNextSequence() {
+ return ++_nextSequence;
+ }
+
+ protected synchronized void addRequest(Request req) {
+ int index = findRequest(req);
+ assert (index < 0) : "How can we get index again? " + index + ":" + req.toString();
+ _requests.add(-index - 1, req);
+ }
+
+ protected void cancel(Request req) {
+ cancel(req.getSequence());
+ }
+
+ protected synchronized void cancel(long seq) {
+ logger.debug(log(seq, "Cancelling."));
+
+ ServerListener listener = _waitForList.remove(seq);
+ if (listener != null) {
+ listener.processDisconnect();
+ }
+ int index = findRequest(seq);
+ if (index >= 0) {
+ _requests.remove(index);
+ }
+ }
+
+ protected synchronized int findRequest(Request req) {
+ return Collections.binarySearch(_requests, req, s_reqComparator);
+ }
+
+ protected synchronized int findRequest(long seq) {
+ return Collections.binarySearch(_requests, seq, s_seqComparator);
+ }
+
+ protected String log(long seq, String msg) {
+ return "Seq " + _name + "-" + seq + ": " + msg;
+ }
+
+ protected void registerListener(long seq, ServerListener listener) {
+ if (logger.isTraceEnabled()) {
+ logger.trace(log(seq, "Registering listener"));
+ }
+ if (listener.getTimeout() != -1) {
+ java.util.concurrent.ScheduledFuture<?> alarmFuture =
+ s_listenerExecutor.schedule(new Alarm(seq), listener.getTimeout(), TimeUnit.SECONDS);
+ _alarmFutures.put(seq, alarmFuture);
+ }
+ _waitForList.put(seq, listener);
+ }
+
+ protected ServerListener unregisterListener(long sequence) {
+ if (logger.isTraceEnabled()) {
+ logger.trace(log(sequence, "Unregistering listener"));
+ }
+ java.util.concurrent.ScheduledFuture<?> alarmFuture = _alarmFutures.remove(sequence);
+ if (alarmFuture != null) {
+ alarmFuture.cancel(false);
+ }
+ return _waitForList.remove(sequence);
+ }
+
+ protected ServerListener getListener(long sequence) {
+ return _waitForList.get(sequence);
+ }
+
+ public String getName() {
+ return _name;
+ }
+
+ public int getQueueSize() {
+ return _requests.size();
+ }
+
+ public boolean processAnswers(long seq, Response resp) {
+ resp.logD("Processing: ", true);
+ boolean processed = false;
+ Answer[] answers = resp.getAnswers();
+ try {
+ ServerListener monitor = getListener(seq);
+ if (monitor == null) {
+ if (answers[0] != null && answers[0].getResult()) {
+ processed = true;
+ }
+ logger.debug(log(seq, "Unable to find listener."));
+ } else {
+ processed = monitor.processAnswers(seq, answers);
+ logger.trace(log(seq, (processed ? "" : " did not ") + " processed "));
+ if (!monitor.isRecurring()) {
+ unregisterListener(seq);
+ }
+ }
+ } finally {
+ // we should always trigger next command execution, even in failure cases - otherwise in exception case
+ // all the remaining will be stuck in the sync queue forever
+ if (resp.executeInSequence()) {
+ sendNext(seq);
+ }
+ }
+ return processed;
+ }
+
+ protected void cancelAllCommands() {
+ for (Iterator<Map.Entry<Long, ServerListener>> it = _waitForList.entrySet().iterator(); it.hasNext(); ) {
+ Map.Entry<Long, ServerListener> entry = it.next();
+ it.remove();
+
+ long seq = entry.getKey();
+ java.util.concurrent.ScheduledFuture<?> alarmFuture = _alarmFutures.remove(seq);
+ if (alarmFuture != null) {
+ alarmFuture.cancel(false);
+ }
+ ServerListener monitor = entry.getValue();
+ logger.debug(log(seq, "Sending disconnect to " + monitor.getClass()));
+ monitor.processDisconnect();
+ }
+ }
+
+ public void cleanup() {
+ cancelAllCommands();
+ _requests.clear();
+ }
+
+ @Override
+ public boolean equals(Object obj) {
+ // Return false straight away.
+ if (obj == null) {
+ return false;
+ }
+ // No need to handle a ClassCastException. If the classes are different, then equals can return false
+ // straight ahead.
+ if (this.getClass() != obj.getClass()) {
+ return false;
+ }
+ ServerAttache that = (ServerAttache) obj;
+ return _name.equals(that.getName());
+ }
+
+ public void send(Request req, ServerListener listener) throws CloudException {
+ long seq = req.getSequence();
+ if (listener != null) {
+ registerListener(seq, listener);
+ }
+
+ synchronized (this) {
+ try {
+ if (isClosed()) {
+ throw new CloudException("The link to the server " + _name + " has been closed", new ClosedChannelException());
+ }
+
+ if (req.executeInSequence() && _currentSequence != null) {
+ req.logD("Waiting for Seq " + _currentSequence + " Scheduling: ", true);
+ addRequest(req);
+ return;
+ }
+
+ // If we got to here either we're not supposed to set the _currentSequence or it is null already.
+ req.logD("Sending ", true);
+ send(req);
+
+ if (req.executeInSequence() && _currentSequence == null) {
+ _currentSequence = seq;
+ logger.trace(log(seq, " is current sequence"));
+ }
+ } catch (CloudException e) {
+ logger.info(log(seq, "Unable to send due to " + e.getMessage()), e);
+ cancel(seq);
+ throw e;
+ } catch (Exception e) {
+ logger.warn(log(seq, "Unable to send due to " + e.getMessage()), e);
+ cancel(seq);
+ throw new CloudException("Problem due to other exception " + e.getMessage(), e);
+ }
+ }
+ }
+
+ public Answer[] send(Request req, int wait) throws CloudException {
+ if (req.getSequence() <= 0) {
+ req.setSequence(getNextSequence());
+ }
+ SynchronousListener sl = new SynchronousListener();
+ sl.setTimeout(wait + 5);
+ send(req, sl);
+
+ long seq = req.getSequence();
+ try {
+ for (int i = 0; i < 2; i++) {
+ Answer[] answers = null;
+ try {
+ answers = sl.waitFor(wait);
+ } catch (InterruptedException e) {
+ logger.debug(log(seq, "Interrupted"));
+ }
+ if (answers != null) {
+ new Response(req, answers).logD("Received: ", false);
+ return answers;
+ }
+
+ answers = sl.getAnswers(); // Try it again.
+ if (answers != null) {
+ new Response(req, answers).logD("Received after timeout: ", true);
+ return answers;
+ }
+
+ Long current = _currentSequence;
+ if (current != null && seq != current) {
+ if (logger.isDebugEnabled()) {
+ logger.debug(log(seq, "Waited too long."));
+ }
+ throw new OperationTimedoutException(req.getCommands(), -1, seq, wait, false);
+ }
+ logger.debug(log(seq, "Waiting some more time because this is the current command"));
+ }
+ throw new OperationTimedoutException(req.getCommands(), -1, seq, wait * 2, true);
+ } catch (OperationTimedoutException e) {
+ logger.warn(log(seq, "Timed out on " + req));
+ cancel(seq);
+ Long current = _currentSequence;
+ if (req.executeInSequence() && (current != null && current == seq)) {
+ sendNext(seq);
+ }
+ throw e;
+ } catch (Exception e) {
+ logger.warn(log(seq, "Exception while waiting for answer"), e);
+ cancel(seq);
+ Long current = _currentSequence;
+ if (req.executeInSequence() && (current != null && current == seq)) {
+ sendNext(seq);
+ }
+ throw new OperationTimedoutException(req.getCommands(), -1, seq, wait, false);
+ } finally {
+ unregisterListener(seq);
+ }
+ }
+
+ protected synchronized void sendNext(long seq) {
+ _currentSequence = null;
+ if (_requests.isEmpty()) {
+ if (logger.isDebugEnabled()) {
+ logger.debug(log(seq, "No more commands found"));
+ }
+ return;
+ }
+
+ Request req = _requests.pop();
+ if (logger.isDebugEnabled()) {
+ logger.debug(log(req.getSequence(), "Sending now. is current sequence."));
+ }
+ try {
+ send(req);
+ } catch (CloudException e) {
+ logger.debug(log(req.getSequence(), "Unable to send the next sequence"));
+ cancel(req.getSequence());
+ }
+ _currentSequence = req.getSequence();
+ }
+
+ public void process(Answer[] answers) {
+ //do nothing
+ }
+
+ /**
+ * sends the request asynchronously.
+ *
+ * @param req
+ * @throws AgentUnavailableException
+ */
+ public synchronized void send(final Request req) throws CloudException {
+ try {
+ _link.send(req.toBytes());
+ } catch (ClosedChannelException e) {
+ throw new CloudException("Channel is closed", e);
+ }
+ }
+
+ /**
+ * Process disconnect.
+ */
+ public void disconnect() {
+ synchronized (this) {
+ logger.debug("Processing Disconnect.");
+ if (_link != null) {
+ _link.close();
+ _link.terminated();
+ }
+ _link = null;
+ }
+ cancelAllCommands();
+ _requests.clear();
+ }
+
+ /**
+ * Is the agent closed for more commands?
+ *
+ * @return true if unable to reach agent or false if reachable.
+ */
+ protected boolean isClosed() {
+ return _link == null;
+ }
+
+ public Link getLink() {
+ return _link;
+ }
+
+ public <T> T send(Long agentId, Command[] commands, Class<T> answerType, int asyncCommandTimeoutSec) throws IOException {
+ String logId = Optional.ofNullable(ThreadContext.get("logcontextid"))
+ .filter(String.class::isInstance)
+ .map(String.class::cast)
+ .orElse(null);
+ if (logId != null) {
+ for (Command command : commands) {
+ if (command.getContextParam("logid") == null) {
+ command.setContextParam("logid", logId);
+ }
+ }
+ }
+ Link link = getLink();
+ String commandName = commands[0].getClass().getSimpleName();
+
+ Long id = Optional.ofNullable(agentId).orElse(-1L);
+ Request request = new Request(id, -1, commands, true, false);
+ Answer[] answers;
+ try {
+ answers = send(request, asyncCommandTimeoutSec);
+ } catch (OperationTimedoutException e) {
+ String msg = String.format("Failed to retrieve answer for command %s to %s", commandName, link);
+ logger.error(msg, e);
+ throw new RuntimeException(msg, e);
+ } catch (CloudException e) {
+ String msg = String.format("Failed to send command %s to %s", commandName, link);
+ logger.error(msg, e);
+ Throwable cause = e.getCause();
+ if (cause instanceof ClosedChannelException) {
+ throw (ClosedChannelException) cause;
+ }
+ throw new RuntimeException(msg, e);
+ }
+ if (ArrayUtils.isEmpty(answers)) {
+ String msg = String.format("Received empty %s response from %s", commandName, link);
+ logger.warn(msg);
+ throw new IllegalArgumentException(msg);
+ }
+ Answer answer = answers[0];
+ String details = answer.getDetails();
+ if (!answer.getResult()) {
+ String msg = String.format("Received unsuccessful %s response status from %s: %s", commandName, link,
+ details);
+ logger.warn(msg);
+ throw new IllegalArgumentException(msg);
+ }
+ String answerName = answer.getClass().getSimpleName();
+ if (!answerType.isInstance(answer)) {
+ String msg = String.format("Received unexpected %s response type %s from %s: %s", commandName,
+ answerName, link, details);
+ logger.warn(msg);
+ throw new IllegalArgumentException(msg);
+ }
+ return answerType.cast(answer);
+ }
+
+ protected class Alarm extends ManagedContextRunnable {
+ long _seq;
+
+ public Alarm(long seq) {
+ _seq = seq;
+ }
+
+ @Override
+ protected void runInContext() {
+ try {
+ ServerListener listener = unregisterListener(_seq);
+ if (listener != null) {
+ cancel(_seq);
+ listener.processTimeout(_seq);
+ }
+ } catch (Exception e) {
+ ServerAttache.logger.warn("Exception ", e);
Review Comment:
got it, thanks
##########
agent/src/main/java/com/cloud/agent/AgentShell.java:
##########
@@ -568,6 +582,7 @@ public static void main(String[] args) {
shell.init(args);
shell.start();
} catch (ConfigurationException e) {
+ LOGGER.fatal(e.getMessage(), e);
Review Comment:
ok that covers it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#discussion_r4144525123
##########
framework/cluster/src/main/java/com/cloud/cluster/ClusterServiceServletAdapter.java:
##########
@@ -84,18 +85,23 @@ protected String getServiceEndpointName(String strPeer) {
if (mshost == null)
return null;
- return composeEndpointName(mshost.getServiceIP(), mshost.getServicePort());
+ return composeEndpointName(mshost, mshost.getServicePort());
}
@Override
public int getServicePort() {
return _clusterServicePort;
}
- private String composeEndpointName(String nodeIP, int port) {
- StringBuffer sb = new StringBuffer();
- sb.append("https://").append(nodeIP).append(":").append(port).append("/clusterservice");
- return sb.toString();
+ private String composeEndpointName(ManagementServerHostVO msHost, int port) {
+ final String hostName = msHost.getName();
+ final String identifier = hostName != null ? hostName : msHost.getServiceIP();
Review Comment:
A blank hostname is treated as usable because only `null` is checked. Legacy or partially populated `mshost` rows can therefore produce `https://:port/clusterservice` even when `serviceIP` is present; use a non-blank check before falling back to the IP.
##########
utils/src/main/java/com/cloud/utils/backoff/impl/ExponentialWithJitterBackoff.java:
##########
@@ -0,0 +1,168 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.utils.backoff.impl;
+
+import com.cloud.utils.DateUtil;
+import com.cloud.utils.NumbersUtil;
+import com.cloud.utils.backoff.BackoffAlgorithm;
+import com.cloud.utils.backoff.BackoffFactory;
+import com.cloud.utils.component.AdapterBase;
+
+import java.security.SecureRandom;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Random;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * Exponential backoff with up/down cycling.
+ * Delay grows exponentially until a maximum, then decreases back to base, then repeats.
+ *
+ * @author mprokopchuk
+ */
+public class ExponentialWithJitterBackoff extends AdapterBase implements BackoffAlgorithm,
+ ExponentialWithJitterBackoffMBean {
+
+ /**
+ * Property name for the minimal delay to be used either by {@code agent.properties} file or by configuration key.
+ */
+ public static final String MIN_DELAY_MS_CONFIG_KEY = "backoff.min_delay_ms";
+
+ /**
+ * Property name for the maximal delay to be used either by {@code agent.properties} file or by configuration key.
+ */
+ public static final String MAX_DELAY_MS_CONFIG_KEY = "backoff.max_delay_ms";
+
+ /**
+ * Default value for minimal delay for the property {@link ExponentialWithJitterBackoff#MIN_DELAY_MS_DEFAULT}.
+ */
+ public static final int MIN_DELAY_MS_DEFAULT = 5_000;
+
+ /**
+ * Default value for maximal delay for the property {@link ExponentialWithJitterBackoff#MAX_DELAY_MS_DEFAULT}.
+ */
+ public static final int MAX_DELAY_MS_DEFAULT = 15_000;
+
+ private final Map<String, Thread> asleep = new ConcurrentHashMap<>();
+ private final Random random = new SecureRandom();
+
+ private int minDelayMs;
+ private int maxDelayMs;
+ private int maxAttempts;
+ private int attemptNumber;
+ private boolean increasing;
+
+ @Override
+ public void waitBeforeRetry() {
+ boolean interrupted = false;
+ long waitMs = getTimeToWait();
+ Thread current = Thread.currentThread();
+ try {
+ asleep.put(current.getName(), current);
+ logger.debug("Going to sleep for {}", DateUtil.formatMillis(waitMs));
+ Thread.sleep(waitMs);
+ logger.debug("Sleep done for {}", DateUtil.formatMillis(waitMs));
+ } catch (InterruptedException e) {
+ interrupted = true;
+ logger.info("Thread {} interrupted while waiting for retry", current.getName(), e);
+ } finally {
+ asleep.remove(current.getName());
+ calculateNextAttempt();
+ if (interrupted) {
+ Thread.currentThread().interrupt();
+ }
+ }
+ }
+
+ /**
+ * Calculates next attempt and direction.
+ */
+ private void calculateNextAttempt() {
+ if (increasing) {
+ int nextAttemptNumber = attemptNumber + 1;
+ increasing = getNextDelay() <= maxDelayMs && nextAttemptNumber <= maxAttempts;
+ if (increasing) {
+ attemptNumber = nextAttemptNumber;
+ }
+ } else {
+ int nextAttemptNumber = Math.max(attemptNumber - 1, 0);
+ increasing = nextAttemptNumber == 0;
+ if (!increasing) {
+ attemptNumber = nextAttemptNumber;
+ }
+ }
+ }
+
+ @Override
+ public void reset() {
+ attemptNumber = 0;
+ }
+
+ @Override
+ public Map<String, String> getConfiguration() {
+ Map<String, String> configuration = new HashMap<>();
+ configuration.put(MIN_DELAY_MS_CONFIG_KEY, String.valueOf(minDelayMs));
+ configuration.put(MAX_DELAY_MS_CONFIG_KEY, String.valueOf(maxDelayMs));
+ configuration.put(BackoffFactory.BACKOFF_IMPLEMENTATION_KEY, this.getClass().getName());
+ return configuration;
+ }
+
+ @Override
+ public boolean configure(String name, Map<String, Object> params) {
+ minDelayMs = NumbersUtil.parseInt((String) params.get(MIN_DELAY_MS_CONFIG_KEY), MIN_DELAY_MS_DEFAULT);
+ maxDelayMs = NumbersUtil.parseInt((String) params.get(MAX_DELAY_MS_CONFIG_KEY), MAX_DELAY_MS_DEFAULT);
+ maxAttempts = (int) Math.round(Math.log((double) maxDelayMs / minDelayMs) / Math.log(2));
+
+ attemptNumber = random.nextInt(maxAttempts + 1);
+ increasing = random.nextBoolean();
+ // do nothing
+ return true;
+ }
+
+ @Override
+ public Collection<String> getWaiters() {
+ return asleep.keySet();
+ }
+
+ @Override
+ public boolean wakeup(String threadName) {
+ Thread th = asleep.get(threadName);
+ if (th != null) {
+ th.interrupt();
+ return true;
+ }
+ return false;
+ }
+
+ private long getNextDelay() {
+ return (long) Math.min(minDelayMs * Math.pow(2, attemptNumber), maxDelayMs);
+ }
+
+ @Override
+ public long getTimeToWait() {
+ long delay = Math.max(1L, getNextDelay());
+ int jitterBound = (int) Math.max(1L, delay / 2);
+ int jitter = random.nextInt(jitterBound);
+ return delay + jitter;
+ }
+
+ @Override
+ public void setTimeToWait(long seconds) {
+ // ignore
+ }
Review Comment:
The MBean exposes `setTimeToWait`, but this implementation silently ignores every value. An operator can successfully invoke the JMX setter and believe the retry delay changed while `getTimeToWait()` remains controlled by the exponential configuration. Either implement the setter's documented management behavior or remove it from this MBean contract.
##########
utils/src/main/java/org/apache/cloudstack/threadcontext/ThreadContextUtil.java:
##########
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package org.apache.cloudstack.threadcontext;
+
+import com.cloud.utils.StringUtils;
+import com.google.gson.Gson;
+import com.google.gson.reflect.TypeToken;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.logging.log4j.ThreadContext;
+
+import java.lang.reflect.Type;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Utility class, helps to propagate {@link ThreadContext} values from parent to child threads.
+ *
+ * @author mprokopchuk
+ */
+public class ThreadContextUtil {
+ private static final Logger logger = LogManager.getLogger(ThreadContextUtil.class);
+
+ public static final String MDC_UUID_KEY = "uuid";
+ public static final String MDC_LOG_CONTEXT_ID_KEY = "logcontextid";
+ public static final String CONTEXT_UUID_KEY = "uuid";
+ public static final String CONTEXT_LOG_ID_KEY = "logid";
+
+ /**
+ * Wrap {@link Runnable} to propagate {@link ThreadContext} values.
+ *
+ * @param delegate
+ * @return
+ */
+ public static Runnable wrapThreadContext(Runnable delegate) {
+ @SuppressWarnings("unchecked")
+ Map<String, String> context = ThreadContext.getContext() != null ?
+ new HashMap<>(ThreadContext.getContext()) : null;
+
+ return () -> {
+ @SuppressWarnings("unchecked")
+ Map<String, String> oldContext = ThreadContext.getContext() != null ?
+ new HashMap<>(ThreadContext.getContext()) : null;
+ try {
+ ThreadContext.clearMap();
+ if (context != null) {
+ context.forEach(ThreadContext::put);
+ }
+ delegate.run();
+ } finally {
+ ThreadContext.clearMap();
+ if (oldContext != null) {
+ oldContext.forEach(ThreadContext::put);
+ }
+ }
+ };
+ }
+
+ /**
+ * Set UUID in ThreadContext.
+ *
+ * @param uuid the UUID value to set
+ */
+ public static void setUuid(String uuid) {
+ if (StringUtils.isNotEmpty(uuid)) {
+ ThreadContext.put(MDC_UUID_KEY, uuid);
+ }
+ }
+
+ /**
+ * Set log context ID in ThreadContext.
+ *
+ * @param logContextId the log context ID value to set
+ */
+ public static void setLogContextId(String logContextId) {
+ if (StringUtils.isNotEmpty(logContextId)) {
+ ThreadContext.put(MDC_LOG_CONTEXT_ID_KEY, logContextId);
+ }
+ }
+
+ /**
+ * Extract UUID from JSON cmdInfo string and set it in MDC if UUID is not already present.
+ * This is specifically used for async job processing.
+ *
+ * @param cmdInfo the JSON string containing command info
+ */
+ public static void extractAndSetUuidFromCmdInfo(String cmdInfo) {
+ if (StringUtils.isBlank((String) ThreadContext.get(MDC_UUID_KEY)) && StringUtils.isNotBlank(cmdInfo)) {
+ try {
+ Type mapType = new TypeToken<Map<String, String>>() {}.getType();
+ Gson gson = new Gson();
+ Map<String, String> params = gson.fromJson(cmdInfo, mapType);
+ String entityUuid = params.get(CONTEXT_UUID_KEY);
Review Comment:
Async VM work jobs do not store `cmdInfo` as a JSON map: `VmWorkSerializer` writes a `JobSerializerHelper` object-serialized string. Parsing every job with Gson therefore fails for those jobs and leaves the UUID out of MDC (while logging a warning), so the claimed UUID propagation does not cover VM async operations. Extract the UUID through the job/VM-work serialization path or use the persisted instance identity instead of assuming JSON.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "winterhazel (via GitHub)" <gi...@apache.org>.
winterhazel commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-6035676594
> @DaanHoogland @nvazquez can we merge this pr or any futher review is needed
@kiranchavala this is a large change on core code. It would be nice to have a few more reviews and approvals before merging. Also, I want to review it too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-6389233485
@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Indirect agent connection improvements [cloudstack]
Posted by "nvazquez (via GitHub)" <gi...@apache.org>.
nvazquez commented on PR #13345:
URL: https://github.com/apache/cloudstack/pull/13345#issuecomment-6389225262
@blueorangutan test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org