You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "jgotteswinter (via GitHub)" <gi...@apache.org> on 2025/05/12 07:46:51 UTC
[I] Agent logs libvirt xml while live migrating vms [cloudstack]
jgotteswinter opened a new issue, #10847:
URL: https://github.com/apache/cloudstack/issues/10847
### problem
When live migrating vms, the whole libvirt xml definition gets logged to /var/log/cloudstack/agent/agent.log (Log Level set to INFO). This includes for example the vnc password, and i guess it would also include the luks password for encrypted volumes.
```
2025-05-09 10:38:46,164 INFO [resource.wrapper.LibvirtMigrateCommandWrapper] (agentRequest-Handler-1:null) (logid:b55d2348) Starting live migration of instance [i-58-xxx-VM] to destination host [qemu+tls://10.xx.xx.xx/system] having the final XML configuration: [<domain type='kvm'>
<name>i-58-4522-VM</name>
<uuid>511d4065-bdd8-4f5f-8dd3-xxxxxxx</uuid>
<description>Ubuntu 22.04 LTS</description>
<memory unit='KiB'>67108864</memory>
<currentMemory unit='KiB'>67108864</currentMemory>
<vcpu placement='static'>16</vcpu>
<iothreads>1</iothreads>
<cputune>
<shares>1250</shares>
<period>10000</period>
<quota>10000</quota>
</cputune>
<resource>
<partition>/machine</partition>
</resource>
<sysinfo type='smbios'>
<system>
<entry name='manufacturer'>Apache Software Foundation</entry>
<entry name='product'>CloudStack KVM Hypervisor</entry>
<entry name='uuid'>511d4065-bdd8-4f5f-8dd3-xxxxxxxx</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='pc-i440fx-8.2'>hvm</type>
<boot dev='cdrom'/>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<cpu mode='host-model' check='partial'>
<topology sockets='4' dies='1' cores='4' threads='1'/>
</cpu>
<clock offset='utc'>
<timer name='kvmclock'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none' io='native' discard='unmap'/>
<source file='/mnt/2c568429-23f5-3e75-8208-xxxxx/0b3ca1d6-bdba-4cc8-bb7f-xxxxx'/>
<target dev='sda' bus='scsi'/>
<serial>0b3ca1d6bdba4xxxxx</serial>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none' io='native' discard='unmap'/>
<source file='/mnt/2c568429-23f5-3e75-8208-xxxxx/cd683178-03ad-4834-b51b-xxxxxx'/>
<target dev='sdb' bus='scsi'/>
<serial>cd68317803ad4834xxx</serial>
<address type='drive' controller='0' bus='0' target='0' unit='1'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<target dev='hdc' bus='ide'/>
<readonly/>
<address type='drive' controller='0' bus='1' target='0' unit='0'/>
</disk>
<controller type='scsi' index='0' model='virtio-scsi'>
<driver queues='16' iothread='1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</controller>
<controller type='ide' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='02:06:01:02:xx:xx'/>
<source bridge='xxxx'/>
<bandwidth>
<inbound average='640000' peak='640000'/>
<outbound average='640000' peak='640000'/>
</bandwidth>
<model type='virtio'/>
<link state='up'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='pty'>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/i-58-xxxx-VM.org.qemu.guest_agent.0'/>
<target type='virtio' name='org.qemu.guest_agent.0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<input type='tablet' bus='usb'>
<address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes' listen='x.x.x.x' passwd='**XYZXYZ**'>
<listen type='address' address='x.x.x.x'/>
</graphics>
<video>
<model type='cirrus' vram='16384' heads='1' primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<watchdog model='i6300esb' action='none'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</watchdog>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</memballoon>
<rng model='virtio'>
<rate bytes='2048' period='1000'/>
<backend model='random'>/dev/random</backend>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</rng>
</devices>
</domain>
].
```
### versions
ACS 4.19.2
### The steps to reproduce the bug
### What to do about it?
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Agent logs libvirt xml while live migrating vms [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland closed issue #10847: Agent logs libvirt xml while live migrating vms
URL: https://github.com/apache/cloudstack/issues/10847
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Agent logs libvirt xml while live migrating vms [cloudstack]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #10847:
URL: https://github.com/apache/cloudstack/issues/10847#issuecomment-3677225783
This issue is stale because it has been open for 120 days with no activity. It may be removed by administrators of this project at any time. Remove the stale label or comment to request for removal of it to prevent this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Agent logs libvirt xml while live migrating vms [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #10847:
URL: https://github.com/apache/cloudstack/issues/10847#issuecomment-3677653105
@jgotteswinter @weizhouapache , for some operators it might even be convenient to have the VNC password available to debug any issues. Do want to keep this issue and if so, how to attack it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Agent logs libvirt xml while live migrating vms [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #10847:
URL: https://github.com/apache/cloudstack/issues/10847#issuecomment-2879201968
> [@weizhouapache](https://github.com/weizhouapache) is this worth an improvement of sorts? If for no other sake then auditing?
IMHO, it is a minor bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Agent logs libvirt xml while live migrating vms [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #10847:
URL: https://github.com/apache/cloudstack/issues/10847#issuecomment-2871394977
indeed,the vnc password is shown when start or migrate the vm
however, to be frank, it is complicated to access the console, even if the password is known
- the host ip must be publibly accessible
- the cloud key/cert must be leaked as well
so I would consider this as a minor issue.
no need to worry about the passphrase of luks-encrypted volumes, or ceph auth.
they are saved in a libvirt secret object, refer to https://libvirt.org/formatsecret.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Agent logs libvirt xml while live migrating vms [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #10847:
URL: https://github.com/apache/cloudstack/issues/10847#issuecomment-3678080423
> [@jgotteswinter](https://github.com/jgotteswinter) [@weizhouapache](https://github.com/weizhouapache) , for some operators it might even be convenient to have the VNC password available to debug any issues. Do want to keep this issue and if so, how to attack it?
@DaanHoogland
well, I think we can mask the vnc password in agent.log and management-server.log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Agent logs libvirt xml while live migrating vms [cloudstack]
Posted by "RosiKyu (via GitHub)" <gi...@apache.org>.
RosiKyu commented on issue #10847:
URL: https://github.com/apache/cloudstack/issues/10847#issuecomment-3765517471
**WIP**
### Test Case 1: VM Start - VNC Password Masking
**Objective** Verify that the VNC password is masked (shown as *****) in the agent log when a VM is started.
**Test Steps**
1. Deploy a new VM (or start a stopped VM) in the CloudStack environment
2. SSH to the KVM host where the VM is running
3. Search the agent log for VM start entries containing VNC graphics configuration:
```
grep -i "Starting.*:" /var/log/cloudstack/agent/agent.log | tail -5
```
4. Search for VNC password entries in the log:
```
grep -E "graphics.*vnc.*passwd" /var/log/cloudstack/agent/agent.log
```
4. Verify no unmasked passwords exist:
```
grep -E "graphics.*vnc.*passwd=['\"][^*]" /var/log/cloudstack/agent/agent.log
```
**Expected Result**
- The VNC password in the logged XML should be masked as `passwd='*****'`
- No unmasked VNC passwords should appear in the agent log
**Actual Result** **PASSED**
- The VNC password is correctly masked as `passwd='*****'`
- No unmasked VNC passwords found in the log
**Evidence**
- VM Start log entry found:
```
2026-01-18 16:57:28,843 DEBUG [resource.wrapper.LibvirtStartCommandWrapper] (AgentRequest-Handler-5:[]) (logid:) Starting r-4-VM : <domain type='kvm'>
```
VNC password masked (grep output):
```
[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm1 ~]# grep -E "graphics.*vnc.*passwd" /var/log/cloudstack/agent/agent.log
<graphics type='vnc' autoport='yes' listen='10.1.34.189' passwd='*****'/>
```
- No unmasked passwords (empty result confirms fix):
```
[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm1 ~]# grep -E "graphics.*vnc.*passwd=['\"][^*]" /var/log/cloudstack/agent/agent.log
```
### Test Case 2: Live Migration - VNC Password Masking.
**Objective** Verify that the VNC password is masked (shown as *****) in the agent log when a VM is live migrated between KVM hosts.
**Test Steps**
1. Identify a running user VM and the host it is currently on
2. Live migrate the VM to another KVM host:
3. After migration completes, SSH to the source host (where VM was before migration) and search for live migration log entries:
```
grep -i "live migration" /var/log/cloudstack/agent/agent.log
```
4. Search for VNC password entries:
```
grep -E "graphics.*vnc.*passwd" /var/log/cloudstack/agent/agent.log
```
5. Check for any unmasked passwords:
```
grep -E "graphics.*vnc.*passwd=['\"][^*]" /var/log/cloudstack/agent/agent.log
```
**Expected Result**
- All VNC passwords in the logged XML should be masked as `passwd='*****'`
- No unmasked VNC passwords should appear in any migration-related log entries
**Actual Result** **PARTIALLY PASSED / BUG FOUND**
- The main migration XML configuration is correctly masked
- However, the `replaceIpForVNCInDescFile` log message **still exposes the actual VNC password**
**Evidence**
- Live migration entry (masked correctly):
```
[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm1 ~]# grep -i "live migration" /var/log/cloudstack/agent/agent.log
2026-01-18 17:05:14,161 INFO [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-3:[]) (logid:) Starting live migration of instance i-2-3-VM to destination host qemu+tls://10.1.33.95/system having the final XML configuration: <domain type='kvm'>
```
- VNC password entries (mixed results):
```
[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm1 ~]# grep -E "graphics.*vnc.*passwd" /var/log/cloudstack/agent/agent.log
<graphics type='vnc' autoport='yes' listen='10.1.34.189' passwd='*****'/>
<graphics type='vnc' autoport='yes' listen='10.1.34.189' passwd='*****'/>
<graphics type='vnc' port='-1' autoport='yes' listen='10.1.34.189' passwd='*****'>
2026-01-18 17:05:14,129 DEBUG [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-3:[]) (logid:)
```
- Replaced the VNC IP address
```
[<graphics type='vnc' port='-1' autoport='yes' listen='10.1.34.189' passwd='UzAp0PQ9'>
</graphics>] with [<graphics type='vnc' port='-1' autoport='yes' listen='10.1.33.95' passwd='UzAp0PQ9'>
<graphics type='vnc' port='-1' autoport='yes' listen='10.1.33.95' passwd='*****'>
```
- Unmasked password found (bug):
```
[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm1 ~]# grep -E "graphics.*vnc.*passwd=['\"][^*]" /var/log/cloudstack/agent/agent.log
2026-01-18 17:05:14,129 DEBUG [resource.wrapper.LibvirtMigrateCommandWrapper] (AgentRequest-Handler-3:[]) (logid:)
```
- Replaced the VNC IP address
```
[<graphics type='vnc' port='-1' autoport='yes' listen='10.1.34.189' passwd='UzAp0PQ9'>
</graphics>] with [<graphics type='vnc' port='-1' autoport='yes' listen='10.1.33.95' passwd='UzAp0PQ9'>
```
**Bug Details**
The `replaceIpForVNCInDescFile` method in `LibvirtMigrateCommandWrapper.java` logs the old and new XML snippets when replacing the VNC IP address, but this log statement is not using the maskSensitiveInfoInXML() function, causing the actual VNC password (UzAp0PQ9) to be exposed in the agent log.
**Affected log message:** "Replaced the VNC IP address [...] with [...]"
### Test Case 3: Console Access Regression Test
**Objective** Verify that the VNC password masking in logs does not break the actual VM console access functionality.
**Test Steps**
1. Open the CloudStack UI
2. Navigate to the test VM (i-2-3-VM)
3. Click on "View Console" button
4. Verify the console window opens and displays the VM screen
5. Verify keyboard input works (login to the VM)
**Expected Result**
- Console should connect successfully
- VM display should be visible
- User should be able to interact with the VM (keyboard/mouse)
**Actual Result: ** **PASSED**
- Console connected successfully via Console Proxy
- VM display visible (CentOS 5.5 login screen)
- Successfully logged in as root - keyboard input working
**Evidence**
<img width="3055" height="1632" alt="Image" src="https://github.com/user-attachments/assets/dea2d094-abb3-46e7-90aa-a8b4b2507e97" />
### Test Case 4: VM Stop and Start - VNC Password Masking
**Objective** Verify that the VNC password remains masked in the agent log when a VM is stopped and then started again.
**Test Steps**
1. Stop the test VM:
2. Wait for the VM to stop completely
3. Start the VM again:
4. Check the agent log on the host where the VM started:
```
grep -i "Starting.*:" /var/log/cloudstack/agent/agent.log | tail -5
grep -E "graphics.*vnc.*passwd" /var/log/cloudstack/agent/agent.log
grep -E "graphics.*vnc.*passwd=['\"][^*]" /var/log/cloudstack/agent/agent.log
```
**Expected Result**
- The VNC password in the logged XML should be masked as `passwd='*****'`
- No unmasked VNC passwords should appear in the agent log
**Actual Result:** **PASSED**
- VM stopped and started successfully on kvm2
- VNC password correctly masked in the start log
**Evidence:**
- VM Start log entry found on kvm2:
```
2026-01-18 17:17:44,269 DEBUG [resource.wrapper.LibvirtStartCommandWrapper] (AgentRequest-Handler-4:[]) (logid:) Starting i-2-3-VM : <domain type='kvm'>
```
- VNC password masked:
```
[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm2 ~]# grep -E "graphics.*vnc.*passwd" /var/log/cloudstack/agent/agent.log
<graphics type='vnc' autoport='yes' listen='10.1.33.95' passwd='*****'/>
```
- No unmasked passwords found:
```
[root@ref-trl-6151-k-Mol9-rositsa-kyuchukova-kvm2 ~]# grep -E "graphics.*vnc.*passwd=['\"][^*]" /var/log/cloudstack/agent/agent.log
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Agent logs libvirt xml while live migrating vms [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #10847:
URL: https://github.com/apache/cloudstack/issues/10847#issuecomment-3768309401
fixed in #12404
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org