You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "jmsperu (via GitHub)" <gi...@apache.org> on 2026/03/11 18:50:09 UTC
[I] KVM agent PostCertificateRenewalTask fails with IllegalStateException during certificate provisioning [cloudstack]
jmsperu opened a new issue, #12795:
URL: https://github.com/apache/cloudstack/issues/12795
## Description
When `provisionCertificate` is called on a KVM host, the agent's `PostCertificateRenewalTask` consistently fails with `java.lang.IllegalStateException: Shutdown in progress`. This causes the host to permanently report `secured=false` in `StartupRoutingCommand.hostDetails`, showing as "Unsecure" in the UI despite having valid TLS certificates and a working SSL connection.
## Steps to Reproduce
1. Add a new KVM host to CloudStack 4.22 with `ca.plugin.root.auth.strictness=true`
2. Run `provisionCertificate hostid=<uuid>`
3. The API returns `{"success": true}` โ keystore, cert, CA cert, and key are all created correctly
4. The `PostCertificateRenewalTask` attempts to restart libvirtd and reconnect, but the cert provisioning triggers an agent restart
5. During the agent shutdown, `Runtime.removeShutdownHook()` throws `IllegalStateException: Shutdown in progress`
6. The agent never sets `secured=true` โ the host permanently shows "Unsecure"
## Error Log (agent.log)
```
INFO [resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper] Restarting libvirt after certificate provisioning/renewal
WARN [resource.wrapper.LibvirtPostCertificateRenewalCommandWrapper] Execution of process for command [sudo service libvirtd restart ] failed.
WARN [cloud.agent.Agent] Failed to execute post certificate renewal command: java.lang.IllegalStateException: Shutdown in progress
at java.base/java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
at java.base/java.lang.Runtime.removeShutdownHook(Runtime.java:244)
at com.cloud.agent.Agent$PostCertificateRenewalTask.runInContext(Agent.java:1377)
```
## Root Cause
In `Agent.java:1377`, the `PostCertificateRenewalTask` calls `Runtime.getRuntime().removeShutdownHook()` during JVM shutdown, which is not allowed per Java spec. The certificate provisioning triggers an agent reconnect/restart, creating a race condition where the PostCertificateRenewal task runs during the shutdown window.
## Impact
- Host permanently shows "Unsecure" in the UI despite valid TLS
- The actual SSL connection works correctly (keystore loads, handshake succeeds)
- The `secured` flag in `host_details` DB table is overwritten to `false` on every agent reconnect
- Manual DB updates are overwritten by the agent's `StartupRoutingCommand`
- Reproduced consistently on multiple `provisionCertificate` attempts
## Suggested Fix
The `PostCertificateRenewalTask.runInContext()` should catch `IllegalStateException` from `removeShutdownHook()` and still proceed with setting the secured flag. Alternatively, check `Thread.currentThread().isInterrupted()` or use a guard flag before calling `removeShutdownHook()`.
```java
// In Agent.java PostCertificateRenewalTask.runInContext()
try {
Runtime.getRuntime().removeShutdownHook(shutdownThread);
} catch (IllegalStateException e) {
// JVM is already shutting down, skip hook removal
LOG.debug("Skipping shutdown hook removal during shutdown", e);
}
```
## Environment
- CloudStack: 4.22.0.0
- OS: Ubuntu 22.04 (also reproduced on fresh provision)
- Java: OpenJDK 11.0.30 and 17.0.18
- KVM/libvirt: working correctly
- `ca.plugin.root.auth.strictness`: true
## Workaround
Manually update the DB: `UPDATE host_details SET value='true' WHERE host_id=<id> AND name='secured';`
This is overwritten on next agent restart but the TLS connection is functionally secure regardless of the flag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] KVM agent PostCertificateRenewalTask fails with IllegalStateException during certificate provisioning [cloudstack]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #12795:
URL: https://github.com/apache/cloudstack/issues/12795#issuecomment-5294166510
## ๐ฏ Triage report
On KVM hosts with `ca.plugin.root.auth.strictness=true`, certificate provisioning triggers an agent restart during which `PostCertificateRenewalTask.runInContext()` throws `IllegalStateException: Shutdown in progress` while removing a shutdown hook. This prevents the `secured` flag from ever being set to `true`, so the host permanently shows "Unsecure" in the UI even though the TLS connection actually works.
### ๐ Assessment
| Dimension | Value | Reasoning |
|---|---|---|
| **Type** | type:bug | Race condition during JVM shutdown causes a persistent, incorrect UI/DB state despite functioning TLS |
| **Component** | component:kvm, component:agent | KVM host agent (`Agent.java`) certificate renewal/reconnect logic |
| **Severity** | Severity:Minor | Cosmetic/status issue only โ TLS connection is functionally secure; DB workaround exists (though it gets overwritten on next restart) |
| **Labels** | type:bug, component:kvm, component:agent, Severity:Minor | |
| **Coding agent** | Suitable | Root cause, exact stack trace, file/line, and a concrete minimal fix (catch `IllegalStateException` around `removeShutdownHook`) are all provided |
### ๐ Similar issues
No duplicate or closely related open issues found.
<details><summary>๐ก Notes and suggestions</summary>
- Consider whether `secured` should also be derived independently from actual TLS handshake success rather than solely relying on this task completing cleanly, to make the state more robust against future race conditions.
- A regression test simulating a shutdown-hook removal during JVM shutdown (mocking `Runtime`) could help prevent recurrence.
</details>
> Generated by [Daily Issue Triage](https://github.com/apache/cloudstack/actions/runs/31806898936) ยท sonnet50 357.5K ยท [โท](https://github.com/search?q=repo%3Aapache%2Fcloudstack+%22gh-aw-workflow-call-id%3A+apache%2Fcloudstack%2Fdaily-issue-triage%22&type=issues)
>
<details>
<summary>Add this agentic workflows to your repo</summary>
To install this agentic workflow, run
```
gh aw add githubnext/agentics/workflows/daily-issue-triage.md@d7c1dc4b72b00607a67caaffdcc216cb64379cf9
```
</details>
<!-- gh-aw-agentic-workflow: Daily Issue Triage, engine: copilot, version: 1.0.52, model: claude-sonnet-5, id: 31806898936, workflow_id: daily-issue-triage, run: https://github.com/apache/cloudstack/actions/runs/31806898936 -->
<!-- gh-aw-workflow-call-id: apache/cloudstack/daily-issue-triage -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org