[I] Retrying host maintenance falsely marks already-migrated VMs as stopped [cloudstack]

["Damans227 (via GitHub)" <gi...@apache.org>]

Damans227 opened a new issue, #14026:
URL: https://github.com/apache/cloudstack/issues/14026

   CLOUDSTACK VERSION
   4.22.1.1, KVM hypervisor
   
   SUMMARY
   Putting a host into maintenance moves its VMs off one at a time, which
   can take a while. Requesting maintenance again on the same host before
   the first request finishes causes it to retry moving every VM still
   listed against that host, including ones already moved. That retry
   fails as expected, but for some VMs it confuses CloudStack into
   thinking the VM was powered off, and it gets marked stopped even though
   it was never touched and kept running the whole time.
   
   EXPECTED
   Requesting maintenance again should not retry VMs already moved.
   
   ACTUAL
   Some already-moved VMs get falsely marked stopped.
   
   NOTE
   The false "stopped" status seems to come from CloudStack losing track
   of which VMs are on which host. That could plausibly happen from causes
   other than this one, but we only have evidence of it via this scenario.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Retrying host maintenance falsely marks already-migrated VMs as stopped [cloudstack]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] commented on issue #14026:
URL: https://github.com/apache/cloudstack/issues/14026#issuecomment-5510499336

   ## 🎯 Triage report
   
   When a second "enter maintenance" request is issued for a KVM host before an earlier in-progress maintenance operation has finished migrating all VMs, CloudStack retries migration for VMs that were already moved. The retry naturally fails (VM is no longer on the host), but in some cases this causes CloudStack to lose track of the VM's actual host and mark it as "stopped" even though it is still running. A maintainer has asked the reporter whether this is a regression and against which version; that has not yet been answered.
   
   ### 📊 Assessment
   
   | Dimension | Value | Reasoning |
   |---|---|---|
   | **Type** | type:bug | Reproducible incorrect state transition (VM falsely marked stopped) |
   | **Component** | component:management-server | Issue concerns host-maintenance orchestration and VM state tracking, which lives in the management server, not the hypervisor agent |
   | **Severity** | Severity:Major | Produces incorrect VM power-state data (false "stopped"), which can affect automation/HA relying on state, but requires a specific race (duplicate maintenance requests) to trigger |
   | **Labels** | type:bug, component:management-server | |
   | **Coding agent** | Needs more info | The maintainer's question about regression/version is still open, and root cause ("losing track of which VMs are on which host") is not yet pinpointed to a specific code path |
   
   ### 🔗 Similar issues
   
   No duplicate or closely related open issues found.
   
   <details><summary>💡 Notes and suggestions</summary>
   
   - Reproduction hinges on a timing race: issuing a second `PrepareForMaintenance`/`maintain` API call while the first is still migrating VMs off the host. Confirming exact API call sequence and CloudStack version comparison (as asked by `@DaanHoogland`) would help narrow this down.
   - Worth checking whether host-maintenance logic properly checks current VM-host assignment before retrying a migration, and whether a failed migration retry incorrectly triggers a VM state update path shared with host-down/fencing handling.
   - Reviewing `HighAvailabilityManager` / `ManagementServerImpl` host-maintenance related VM state transitions and any place that marks a VM `Stopped` in response to a migration failure would be a good starting point.
   
   </details>
   
   
   
   > Generated by [Daily Issue Triage](https://github.com/apache/cloudstack/actions/runs/33636271985) · sonnet50 135.6K · [◷](https://github.com/search?q=repo%3Aapache%2Fcloudstack+%22gh-aw-workflow-call-id%3A+apache%2Fcloudstack%2Fdaily-issue-triage%22&type=issues)
   >
   <details>
   <summary>Add this agentic workflows to your repo</summary>
   
   To install this agentic workflow, run
   
   ```
   gh aw add githubnext/agentics/workflows/daily-issue-triage.md@d7c1dc4b72b00607a67caaffdcc216cb64379cf9
   ```
   </details>
   
   
   <!-- gh-aw-agentic-workflow: Daily Issue Triage, engine: copilot, version: 1.0.52, model: claude-sonnet-5, id: 33636271985, workflow_id: daily-issue-triage, run: https://github.com/apache/cloudstack/actions/runs/33636271985 -->
   <!-- gh-aw-workflow-call-id: apache/cloudstack/daily-issue-triage -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Retrying host maintenance falsely marks already-migrated VMs as stopped [cloudstack]

["DaanHoogland (via GitHub)" <gi...@apache.org>]

DaanHoogland commented on issue #14026:
URL: https://github.com/apache/cloudstack/issues/14026#issuecomment-5505771200

   @Damans227 is this a regression, and in comparison to what version?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org