You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "vishesh92 (via GitHub)" <gi...@apache.org> on 2026/10/13 08:31:52 UTC
[I] SQLIntegrityConstraintViolationException for SystemVMs after a network disruption [cloudstack]
vishesh92 opened a new issue, #14391:
URL: https://github.com/apache/cloudstack/issues/14391
### problem
If a host's network is unreachable and the systemVM is stopped and removed, when the host comes back up again, I see the below errors in the log and the running systemVMs are not shown in the UI.
```
2026-10-13 08:27:29,001 ERROR [c.c.h.d.HostDaoImpl] (AgentConnectTaskPool-93:[ctx-e2dc4381]) (logid:f95ffe51) DB Exception on: HikariProxyPreparedStatement@140904250 wrapping com.mysql.cj.jdbc.ServerPreparedStatement[233]: INSERT INTO host (host.disconnected, host.name, host.status, host.type, host.private_ip_address, host.private_mac_address, host.private_netmask, host.public_netmask, host.public_ip_address, host.public_mac_address, host.storage_ip_address, host.cluster_id, host.storage_netmask, host.storage_mac_address, host.storage_ip_address_2, host.storage_netmask_2, host.storage_mac_address_2, host.hypervisor_type, host.proxy_port, host.resource, host.fs_type, host.available, host.setup, host.resource_state, host.hypervisor_version, host.arch, host.update_count, host.uuid, host.storage_access_groups, host.data_center_id, host.pod_id, host.cpu_sockets, host.cpus, host.url, host.speed, host.ram, host.parent, host.guid, host.capabilities, host.total_size, host.last_ping, host.m
gmt_server_id, host.last_mgmt_server_id, host.dom0_memory, host.version, host.created) VALUES (NULL, x'732d31342d564d', 'Creating', 'SecondaryStorageVM', x'31302e302e33362e3637', x'31653a30313a37303a30303a30303a3162', x'3235352e3235352e3234302e30', x'3235352e3235352e3234302e30', x'31302e302e35322e3631', x'31653a30313a36613a30303a30303a3031', x'31302e302e33362e3637', NULL, x'3235352e3235352e3234302e30', x'31653a30313a37303a30303a30303a3162', NULL, NULL, NULL, NULL, NULL, NULL, NULL, 1, 0, 'Creating', NULL, 'x86_64', 0, x'30656439363763342d383733362d343638652d626130322d393337303762393030646364', NULL, 1, 1, NULL, NULL, x'4e6f49716e', NULL, 0, NULL, x'732d31342d564d2d5072656d69756d5365636f6e6461727953746f726167655265736f75726365', NULL, NULL, 1749882860, 32988771385657, NULL, 0, x'342e32322e312e31', '2026-10-13 08:27:29') java.sql.SQLIntegrityConstraintViolationException: Duplicate entry 's-14-VM-PremiumSecondaryStorageResource' for key 'host.guid'
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:118)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
at com.mysql.cj.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:555)
at com.mysql.cj.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:339)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1061)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1009)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1320)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:994)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
at com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1639)
at com.cloud.host.dao.HostDaoImpl.persist(HostDaoImpl.java:1005)
at com.cloud.host.dao.HostDaoImpl.persist(HostDaoImpl.java:85)
at jdk.internal.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at jdk.proxy3/jdk.proxy3.$Proxy112.persist(Unknown Source)
at com.cloud.resource.ResourceManagerImpl.persistNewHost(ResourceManagerImpl.java:3316)
at com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:3286)
at com.cloud.resource.ResourceManagerImpl.createHostVOForConnectedAgent(ResourceManagerImpl.java:3633)
at jdk.internal.reflect.GeneratedMethodAccessor257.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at jdk.proxy3/jdk.proxy3.$Proxy217.createHostVOForConnectedAgent(Unknown Source)
at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1438)
at com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1538)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
2026-10-13 08:27:29,002 DEBUG [c.c.u.d.T.Transaction] (AgentConnectTaskPool-93:[ctx-e2dc4381]) (logid:f95ffe51) Rolling back the transaction: Time = 2 Name = AgentConnectTaskPool-93; called by -TransactionLegacy.rollback:905-TransactionLegacy.removeUpTo:848-TransactionLegacy.close:672-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:175-ExposeInvocationInterceptor.invoke:97-ReflectiveMethodInvocation.proceed:186-JdkDynamicAopProxy.invoke:215-$Proxy112.persist:-1-ResourceManagerImpl.persistNewHost:3316-ResourceManagerImpl.createHostVO:3286-ResourceManagerImpl.createHostVOForConnectedAgent:3633
2026-10-13 08:27:29,002 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-93:[ctx-e2dc4381]) (logid:f95ffe51) Failed to handle host connection: javax.persistence.EntityExistsException: Entity already exists
at com.cloud.utils.db.GenericDaoBase.handleEntityExistsException(GenericDaoBase.java:892)
at com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1678)
at com.cloud.host.dao.HostDaoImpl.persist(HostDaoImpl.java:1005)
at com.cloud.host.dao.HostDaoImpl.persist(HostDaoImpl.java:85)
at jdk.internal.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at jdk.proxy3/jdk.proxy3.$Proxy112.persist(Unknown Source)
at com.cloud.resource.ResourceManagerImpl.persistNewHost(ResourceManagerImpl.java:3316)
at com.cloud.resource.ResourceManagerImpl.createHostVO(ResourceManagerImpl.java:3286)
at com.cloud.resource.ResourceManagerImpl.createHostVOForConnectedAgent(ResourceManagerImpl.java:3633)
at jdk.internal.reflect.GeneratedMethodAccessor257.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at jdk.proxy3/jdk.proxy3.$Proxy217.createHostVOForConnectedAgent(Unknown Source)
at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1438)
at com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1538)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.sql.SQLIntegrityConstraintViolationException: Duplicate entry 's-14-VM-PremiumSecondaryStorageResource' for key 'host.guid'
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:118)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
at com.mysql.cj.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:555)
at com.mysql.cj.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:339)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1061)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1009)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1320)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:994)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
at com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1639)
... 37 more
```
### versions
most likely all
### The steps to reproduce the bug
1. Block all traffic to the Management Servers
```sh
MS_IPS="10.0.0.10 10.0.0.11"
modprobe br_netfilter
sysctl -w net.bridge.bridge-nf-call-iptables=1
for ip in $MS_IPS; do
iptables -I OUTPUT -d "$ip" -j DROP # cloudstack-agent -> MS
iptables -I INPUT -s "$ip" -j DROP # MS -> host
iptables -I FORWARD -d "$ip" -j DROP # system VMs -> MS (bridged)
iptables -I FORWARD -s "$ip" -j DROP # MS -> system VMs
done
```
2. Force Stop the systemVM running on the host via the UI and then destroy it from the UI. Now the systemVM is marked as removed in the management server but the VM is still running. New systemVMs should start coming up after this.
3. Flush the iptables `iptables -F` on the host to restore connectivity with the management server
4. After this, you will see the above error in the logs.
### What to do about it?
Remove the unique constraint or recover the older (removed) row and update that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org