You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "Vash_X@gmx.de" <Va...@gmx.de> on 2022/08/28 11:16:46 UTC
Error migrate System VMs after upgrageing to 4.17
Hi everyone,
faceing some challanges again after upgradeing to 4.17.
Did as explained in the docs.
Afterwards I am now not able to migrate system VMs to different hosts.
Getting the error
"Exception during migrate: org.libvirt.LibvirtException: Cannot access
storage file '/usr/share/cloudstack-common/vms/systemvm.iso': No such file
or directory"
Checked the exitstence of the file on the servers:
Mangement-Server has a "systemvm.iso.bak"
Hosts - no "systemvm.iso" at all present.
Maybe someone has an idea on how to move on with this?
Regards,
Chris
Re: Error migrate System VMs after upgrageing to 4.17
Posted by "Vash_X@gmx.de" <Va...@gmx.de>.
Hi Nicolas,
here the link to the the newly created issue at github
https://github.com/apache/cloudstack/issues/6698
Regards,
Chris
Am Sa., 3. Sept. 2022 um 15:20 Uhr schrieb Nicolas Vazquez <
Nicolas.Vazquez@shapeblue.com>:
> Thanks Chris,
>
> Can you please open a Github issue for this problem?
>
> Regards,
> Nicolas Vazquez
>
>
>
>
>
> ------------------------------
> *From:* Vash_X@gmx.de <Va...@gmx.de>
> *Sent:* Friday, September 2, 2022 7:15:15 PM
> *To:* users@cloudstack.apache.org <us...@cloudstack.apache.org>; Slavka
> Peleva <sl...@storpool.com>; Wei ZHOU <us...@gmail.com>
> *Subject:* Re: Error migrate System VMs after upgrageing to 4.17
>
> Hi,
>
> time for a short update.
>
> Today i tried my luck by switching the global setting
> *system.vm.public.ip.reservation.mode.strictness* from true to false.
> Afterwards i was able to create a systemvm and migrate it to another
> server. --> @Slavka Peleva <sl...@storpool.com> so the initial "problem"
> of not finding systemvm.iso seems to be solved.
>
> The used public IP "belongs" to a ip range reserved for the domain "Root" (
> [Root] in WebGUI Style) which is not marked for use with systemVMs.
>
> As much as i like to have the system up running again, still the question
> is why the reservation marked for usage of systemVMs - belonging to the
> account system as the GUI says - isn't used anymore?
>
> PS.:
> I tried to create a new, additional ip range wich is marked as "Reserved
> for SystemVMs" for the domain root. Afterwards I enabled the global setting
> as described to "true".
> Afterwards i distroyed a consoleproxy. The deployment of the new proxy
> fails due to "insufficent address capacity". So it seems the IP Range is
> not "respected"?
>
> I checked the table "domain_vlan_map", which is used to get the vlan
> information and there is NO entry of the newly created ip range...
>
> regards
> Christopher
>
> Am Do., 1. Sept. 2022 um 01:01 Uhr schrieb Vash_X@gmx.de <Va...@gmx.de>:
>
> > Hi Slavka,
> >
> > be prepared vor a longer answer as this seems to be a more specific
> > problem - but first let me thank you for your guidance and support!
> >
> > > The discussion started about a problem migrating the system VMs, and
> > probably many other issues will appear.
> > Don't give up :-D I might see some light ahead....
> >
> > Bevore getting into the logs and my findings let me answer you your
> > questions briefly.
> > > Can you tell from which version you started the upgrade?
> > I started from 4.16.1; the installation was builded with 4.15.1 and then
> > updated to 4.16 / 4.16.1
> > > Did the upgrade was successful for the management/agents?
> > The upgrade was successfull - even if we had our issues which are
> > currently in a ticket at github
> > https://github.com/apache/cloudstack/issues/6680
> > The upgrade of the management server was successfully performed too.
> > > Did you restart the agents on all machines?
> > We did as we had trouble with the changed detection of CPU speeds on the
> > host.
> > The whole "problem" of the SystemVM.iso is at least from my point of view
> > no problem of the agent-update.
> > a) we had the expection from reading the upgradeguide, that a "simple"
> > patching of the systemVMs would be enough. That at least was our
> > understanding of the "matrix" showing the patching capabilities of the
> > different versions / "upgrade paths".
> > b) currently we assume, that the main-reason for the failing migration
> is,
> > that the SSVM / CPVM are still running without a clear "restart". Which
> we
> > can't test as we still have to solve the problem of the public IP
> > allocation.
> > > Can you share the content in the directory
> > `/usr/share/cloudstack-agent/lib/` from the hypervisor where the
> migration
> > failed?
> >
> > activation-1.1.1.jar
> > cloud-framework-events-4.17.0.1.jar
> > error_prone_annotations-2.3.4.jar jaxb-impl-2.3.0.jar
> > okio-2.8.0.jar
> > annotations-13.0.jar
> > cloud-framework-ipc-4.17.0.1.jar esapi-2.1.0.1.jar
> > jbcrypt-1.0.0.jar
> > opensaml-2.6.6.jar
> > antisamy-1.5.3.jar
> > cloud-framework-jobs-4.17.0.1.jar failureaccess-1.0.1.jar
> > jcl-over-slf4j-1.7.5.jar
> > openws-1.5.6.jar
> > aopalliance-repackaged-2.5.0-b42.jar
> > cloud-framework-managed-context-4.17.0.1.jar groovy-all-2.4.17.jar
> > jersey-client-2.26.jar
> > osgi-resource-locator-1.0.1.jar
> > apache-log4j-extras-1.2.17.jar
> > cloud-framework-rest-4.17.0.1.jar gson-1.7.2.jar
> > jersey-common-2.26.jar
> > protobuf-java-3.11.1.jar
> > aspectjweaver-1.8.13.jar
> > cloud-framework-security-4.17.0.1.jar guava-30.1-jre.jar
> > jersey-entity-filtering-2.26.jar
> > rados-0.6.0.jar
> > aws-java-sdk-core-1.11.940.jar
> > cloud-plugin-hypervisor-kvm-4.17.0.1.jar guava-testlib-18.0.jar
> > jersey-hk2-2.26.jar
> > reflections-0.9.12.jar
> > aws-java-sdk-kms-1.11.940.jar
> > cloud-plugin-network-ovs-4.17.0.1.jar hk2-api-2.5.0-b42.jar
> > jersey-media-json-jackson-2.26.jar
> > reload4j-1.2.20.jar
> > aws-java-sdk-s3-1.11.940.jar
> > cloud-plugin-storage-volume-scaleio-4.17.0.1.jar
> hk2-locator-2.5.0-b42.jar
> > jersey-media-multipart-2.26.jar
> > retrofit-2.9.0.jar
> > batik-constants-1.14.jar
> > cloud-secondary-storage-4.17.0.1.jar hk2-utils-2.5.0-b42.jar
> > jmespath-java-1.11.940.jar
> > rolling-maintenance
> > batik-css-1.14.jar cloud-server-4.17.0.1.jar
> > httpclient-4.5.13.jar
> > jna-5.5.0.jar
> > slf4j-api-1.7.5.jar
> > batik-i18n-1.14.jar cloud-utils-4.17.0.1.jar
> > httpcore-4.4.14.jar
> > jna-platform-5.5.0.jar
> > spring-aop-5.3.18.jar
> > batik-shared-resources-1.14.jar
> commons-beanutils-1.9.2.jar
> > influxdb-java-2.21.jar
> > joda-time-2.10.9.jar
> > spring-beans-5.3.18.jar
> > batik-util-1.14.jar commons-codec-1.15.jar
> > ini4j-0.5.4.jar
> > jsch-0.1.55.jar
> > spring-context-5.3.18.jar
> > bcpkix-jdk15on-1.70.jar
> > commons-collections-3.2.1.jar ion-java-1.0.2.jar
> > json-20090211.jar
> > spring-core-5.3.18.jar
> > bcprov-jdk15on-1.70.jar commons-compress-1.21.jar
> > j2objc-annotations-1.3.jar
> > jsr305-1.3.9.jar
> > spring-expression-5.3.18.jar
> > bctls-jdk15on-1.70.jar
> > commons-configuration-1.10.jar
> > jackson-annotations-2.12.5.jar jul-to-slf4j-1.7.5.jar
> > spring-jcl-5.3.18.jar
> > bcutil-jdk15on-1.70.jar commons-daemon-1.2.3.jar
> > jackson-core-2.12.5.jar
> > kotlin-stdlib-1.4.10.jar
> > spring-web-5.3.18.jar
> > bsh-core-2.0b4.jar commons-dbcp2-2.8.0.jar
> > jackson-databind-2.12.5.jar
> > kotlin-stdlib-common-1.4.0.jar
> > stax2-api-3.1.4.jar
> > cglib-nodep-3.3.0.jar commons-digester-1.8.1.jar
> > jackson-dataformat-cbor-2.6.7.jar
> > kotlin-stdlib-jdk7-1.4.10.jar
> > swagger-annotations-2.0.0.jar
> > checker-qual-3.5.0.jar commons-email-1.5.jar
> > jackson-datatype-jsr310-2.12.5.jar
> > kotlin-stdlib-jdk8-1.4.10.jar
> > tink-1.5.0.jar
> > cloud-agent-4.17.0.1.jar commons-fileupload-1.4.jar
> > jackson-jaxrs-base-2.12.1.jar
> > libvirt-0.5.2.jar
> > trilead-ssh2-build-217-jenkins-27.jar
> > cloud-api-4.17.0.1.jar commons-httpclient-3.1.jar
> > jackson-jaxrs-json-provider-2.12.1.jar
> > libvirtqemuhook
> > truth-0.23.jar
> > cloud-core-4.17.0.1.jar commons-io-2.8.0.jar
> > jackson-module-jaxb-annotations-2.8.4.jar
> > listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
> > velocity-1.7.jar
> > cloud-engine-api-4.17.0.1.jar commons-lang-2.6.jar
> > jasypt-1.9.3.jar
> > log4j-over-slf4j-1.7.5.jar
> > woodstox-core-5.0.3.jar
> > cloud-engine-components-api-4.17.0.1.jar commons-lang3-3.11.jar
> > java-ipv6-0.17.jar
> > logging-interceptor-4.9.0.jar
> > xalan-2.7.0.jar
> > cloud-engine-schema-4.17.0.1.jar commons-logging-1.2.jar
> > java-linstor-0.3.0.jar
> > mail-1.5.0-b01.jar
> > xercesImpl-2.8.0.jar
> > cloud-engine-storage-4.17.0.1.jar commons-math3-3.6.1.jar
> > javassist-3.27.0-GA.jar
> > metrics-core-3.0.2.jar
> > xmlgraphics-commons-2.6.jar
> > cloud-engine-storage-configdrive-4.17.0.1.jar commons-net-3.7.2.jar
> > javax.annotation-api-1.3.2.jar
> > metrics-jvm-3.0.2.jar
> > xmlpull-1.1.3.1.jar
> > cloud-engine-storage-image-4.17.0.1.jar commons-pool2-2.9.0.jar
> > javax.inject-1.jar
> > mimepull-1.9.6.jar
> > xmlschema-core-2.2.5.jar
> > cloud-engine-storage-volume-4.17.0.1.jar commons-validator-1.6.jar
> > javax.inject-2.5.0-b42.jar
> > moshi-1.8.0.jar
> > xmlsec-1.5.7.jar
> > cloud-framework-agent-lb-4.17.0.1.jar converter-moshi-2.9.0.jar
> > javax.mail-1.5.6.jar
> > msgpack-core-0.8.21.jar
> > xmltooling-1.4.6.jar
> > cloud-framework-ca-4.17.0.1.jar cxf-core-3.2.14.jar
> > javax.persistence-2.2.1.jar
> > mxparser-1.2.2.jar
> > xom-1.2.5.jar
> > cloud-framework-cluster-4.17.0.1.jar
> > cxf-rt-frontend-jaxrs-3.2.14.jar
> > javax.servlet-api-4.0.1.jar nekohtml-1.9.16.jar
> > xstream-1.4.19.jar
> > cloud-framework-config-4.17.0.1.jar
> > cxf-rt-transports-http-3.2.14.jar
> javax.ws.rs-api-2.1.jar
> > netty-all-4.1.42.Final.jar
> > cloud-framework-db-4.17.0.1.jar eddsa-0.3.0.jar
> > jaxb-api-2.3.0.jar
> > not-yet-commons-ssl-0.3.9.jar
> > cloud-framework-direct-download-4.17.0.1.jar ehcache-core-2.6.11.jar
> > jaxb-core-2.3.0.jar
> >
> > And now to the problem of the ip's....
> > Thanks for your guidance, i guess i can give some more information. For
> > the findings to understand i guess i need to elaborate a bit on the
> current
> > configuration.
> >
> > Currently we are having only one Zone up and running (ID=1).
> > In this Zone we have 3 different domains which are "Root","A","B".
> > Each domain has a account linked: Root = Admin / A = AccA / B = AccB.
> > The public network of the zone has currently 3 ip ranges which are:
> [Root]
> > system (System VMs = true) / [Root] / [B]AccB (written like in the WebGUI
> > which is i guess going to be important for understanding).
> > This kind of configuration has been in place since the initial setup and
> > has been working without problems.
> >
> > Afterwards you will find the logs with the database querys - i started
> > with a deleting the stopped consoleproxy. i will only share the part when
> > the process of configuring the NIC / IPs for the console proxy will take
> > place.
> > I put some statements into it - for me to get a better understanding of
> > the process - and i hope that you will get through this a bit better. The
> > comments are referenceing on the following "closing" statements. How
> ever i
> > didn't deleted anything.
> >
> >
> > ######################################################################
> Nic
> > allocation
> >
> > 2022-08-31 18:44:21,203 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Allocating nics for VM
> > instance {id: "86", name: "v-86-VM", uuid:
> > "6f77acb4-c65e-49e8-ae03-1175b3974412", type="ConsoleProxy"}
> > 2022-08-31 18:44:21,203 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> > consoleproxy-1 : tx-79, consoleproxy-1,
> > 2022-08-31 18:44:21,204 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: start requested by:
> >
> -Transaction.execute:39-NetworkOrchestrator.allocate:778-VirtualMachineManagerImpl$1.doInTransactionWithoutResult:478-TransactionCallbackWithExceptionNoReturn.doInTransaction:25-TransactionCallbackWithExceptionNoReturn.doInTransaction:21-Transaction.execute:40-VirtualMachineManagerImpl.allocate:469-VirtualMachineManagerImpl.allocate:528-ConsoleProxyManagerImpl.createProxyInstance:700-ConsoleProxyManagerImpl.startNew:572-ConsoleProxyManagerImpl.allocCapacity:809-ConsoleProxyManagerImpl.expandPool:1537
> > 2022-08-31 18:44:21,204 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: has already been
> > started.
> > 2022-08-31 18:44:21,204 TRACE [o.a.c.e.o.NetworkOrchestrator]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) allocating networks for
> > v-86-VM(template c20799e4-a990-449a-aae6-4b67374c7aff); 3 networks
> > 2022-08-31 18:44:21,204 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> > consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
> > 2022-08-31 18:44:21,206 TRACE [c.c.u.d.T.Connection]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Creating a DB connection
> > with txn: for 0: dbconn623913035. Stack:
> >
> -TransactionLegacy.prepareStatement:466-TransactionLegacy.prepareAutoCloseStatement:459-GenericDaoBase.searchIncludingRemoved:396-GenericDaoBase.searchIncludingRemoved:359-GenericDaoBase.search:348-GenericDaoBase.search:1319-TemplateDeployAsIsDetailsDaoImpl.listDetailsByTemplateIdMatchingPrefix:64-TemplateDeployAsIsDetailsDaoImpl.listNetworkRequirementsByTemplateId:69-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62-DelegatingMethodAccessorImpl.invoke:43-Method.invoke:566
> > 2022-08-31 18:44:21,206 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> > template_deploy_as_is_details.id,
> > template_deploy_as_is_details.template_id,
> > template_deploy_as_is_details.name, template_deploy_as_is_details.value
> > FROM template_deploy_as_is_details WHERE
> > template_deploy_as_is_details.template_id = ? AND
> > template_deploy_as_is_details.name LIKE ?
> >
> > #### template_deploy_as_is_details = empty --> no results returned
> >
> > 2022-08-31 18:44:21,206 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> > com.mysql.cj.jdbc.ClientPreparedStatement: SELECT
> > template_deploy_as_is_details.id,
> > template_deploy_as_is_details.template_id,
> > template_deploy_as_is_details.name, template_deploy_as_is_details.value
> > FROM template_deploy_as_is_details WHERE
> > template_deploy_as_is_details.template_id = 213 AND
> > template_deploy_as_is_details.name LIKE x'6E6574776F726B2D25'
> > 2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Connection]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: Not closing DB
> > connection because we're still in a transaction.
> > 2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> > consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
> > 2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> > connection
> > 2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> > networks.id, networks.mode, networks.broadcast_domain_type,
> > networks.traffic_type, networks.name, networks.display_text,
> > networks.broadcast_uri, networks.gateway, networks.cidr,
> > networks.network_cidr, networks.network_offering_id, networks.vpc_id,
> > networks.physical_network_id, networks.data_center_id, networks.related,
> > networks.guru_name, networks.state, networks.redundant, networks.dns1,
> > networks.domain_id, networks.account_id, networks.set_fields,
> > networks.guru_data, networks.dns2, networks.network_domain,
> > networks.removed, networks.created, networks.reservation_id,
> networks.uuid,
> > networks.guest_type, networks.acl_type, networks.restart_required,
> > networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr,
> > networks.display_network, networks.network_acl_id, networks.streched_l2,
> > networks.external_id FROM networks WHERE networks.id = ? AND
> > networks.removed IS NULL
> > 2022-08-31 18:44:21,209 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Allocating nic for vm VM
> > instance {id: "86", name: "v-86-VM", uuid:
> > "6f77acb4-c65e-49e8-ae03-1175b3974412", type="ConsoleProxy"} in network
> > Network {"id": 200, "name": "null", "uuid":
> > "1b0bf965-a391-478d-833d-3e37e4a15eb8", "networkofferingid": 1} with
> > requested profile NicProfile
> >
> {"broadcastUri":null,"iPv4Address":null,"id":0,"reservationId":null,"vmId":0}
> > 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> > consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
> > 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> > connection
> > 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> > data_center.id, data_center.name, data_center.description,
> > data_center.dns1, data_center.dns2, data_center.ip6_dns1,
> > data_center.ip6_dns2, data_center.internal_dns1,
> data_center.internal_dns2,
> > data_center.router_mac_address, data_center.guest_network_cidr,
> > data_center.domain_id, data_center.domain, data_center.networktype,
> > data_center.dns_provider, data_center.dhcp_provider,
> > data_center.gateway_provider, data_center.vpn_provider,
> > data_center.userdata_provider, data_center.lb_provider,
> > data_center.firewall_provider, data_center.mac_address,
> > data_center.zone_token, data_center.removed,
> data_center.allocation_state,
> > data_center.uuid, data_center.is_security_group_enabled,
> > data_center.is_local_storage_enabled, data_center.sort_key FROM
> data_center
> > WHERE data_center.id = ? AND data_center.removed IS NULL
> >
> > ### Allocating public nic
> >
> > 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> > com.mysql.cj.jdbc.ClientPreparedStatement: SELECT networks.id,
> > networks.mode, networks.broadcast_domain_type, networks.traffic_type,
> > networks.name, networks.display_text, networks.broadcast_uri,
> > networks.gateway, networks.cidr, networks.network_cidr,
> > networks.network_offering_id, networks.vpc_id,
> > networks.physical_network_id, networks.data_center_id, networks.related,
> > networks.guru_name, networks.state, networks.redundant, networks.dns1,
> > networks.domain_id, networks.account_id, networks.set_fields,
> > networks.guru_data, networks.dns2, networks.network_domain,
> > networks.removed, networks.created, networks.reservation_id,
> networks.uuid,
> > networks.guest_type, networks.acl_type, networks.restart_required,
> > networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr,
> > networks.display_network, networks.network_acl_id, networks.streched_l2,
> > networks.external_id FROM networks WHERE networks.id = 200 AND
> > networks.removed IS NULL
> > 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> > consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
> > 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> > connection
> > 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> > account.id, account.account_name, account.type, account.role_id,
> > account.domain_id, account.state, account.created, account.removed,
> > account.cleanup_needed, account.network_domain, account.uuid,
> > account.default_zone_id, account.default FROM account WHERE account.id =
> > ? AND account.removed IS NULL
> >
> > ### Getting Zone Information
> >
> > 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> > com.mysql.cj.jdbc.ClientPreparedStatement: SELECT data_center.id,
> > data_center.name, data_center.description, data_center.dns1,
> > data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2,
> > data_center.internal_dns1, data_center.internal_dns2,
> > data_center.router_mac_address, data_center.guest_network_cidr,
> > data_center.domain_id, data_center.domain, data_center.networktype,
> > data_center.dns_provider, data_center.dhcp_provider,
> > data_center.gateway_provider, data_center.vpn_provider,
> > data_center.userdata_provider, data_center.lb_provider,
> > data_center.firewall_provider, data_center.mac_address,
> > data_center.zone_token, data_center.removed,
> data_center.allocation_state,
> > data_center.uuid, data_center.is_security_group_enabled,
> > data_center.is_local_storage_enabled, data_center.sort_key FROM
> data_center
> > WHERE data_center.id = 1 AND data_center.removed IS NULL
> > 2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> > consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
> > 2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: start requested by:
> >
> -Transaction.execute:39-IpAddressManagerImpl.listAvailablePublicIps:824-IpAddressManagerImpl.fetchNewPublicIp:811-IpAddressManagerImpl.fetchNewPublicIp:803-IpAddressManagerImpl.assignPublicIpAddress:784-PublicNetworkGuru.getIp:125-PublicNetworkGuru.allocate:172-NetworkOrchestrator.allocateNic:973-NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice:823-NetworkOrchestrator$3.doInTransactionWithoutResult:797-TransactionCallbackWithExceptionNoReturn.doInTransaction:25-TransactionCallbackWithExceptionNoReturn.doInTransaction:21
> > 2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: has already been
> > started.
> > 2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> > consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> > 2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> > connection
> > 2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> > data_center.id, data_center.name, data_center.description,
> > data_center.dns1, data_center.dns2, data_center.ip6_dns1,
> > data_center.ip6_dns2, data_center.internal_dns1,
> data_center.internal_dns2,
> > data_center.router_mac_address, data_center.guest_network_cidr,
> > data_center.domain_id, data_center.domain, data_center.networktype,
> > data_center.dns_provider, data_center.dhcp_provider,
> > data_center.gateway_provider, data_center.vpn_provider,
> > data_center.userdata_provider, data_center.lb_provider,
> > data_center.firewall_provider, data_center.mac_address,
> > data_center.zone_token, data_center.removed,
> data_center.allocation_state,
> > data_center.uuid, data_center.is_security_group_enabled,
> > data_center.is_local_storage_enabled, data_center.sort_key FROM
> data_center
> > WHERE data_center.id = ? AND data_center.removed IS NULL
> >
> > ### Account Information "system"
> >
> > 2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> > com.mysql.cj.jdbc.ClientPreparedStatement: SELECT account.id,
> > account.account_name, account.type, account.role_id, account.domain_id,
> > account.state, account.created, account.removed, account.cleanup_needed,
> > account.network_domain, account.uuid, account.default_zone_id,
> > account.default FROM account WHERE account.id = 1 AND account.removed
> IS
> > NULL
> > 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> > consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> > 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> > connection
> > 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> > networks.id, networks.mode, networks.broadcast_domain_type,
> > networks.traffic_type, networks.name, networks.display_text,
> > networks.broadcast_uri, networks.gateway, networks.cidr,
> > networks.network_cidr, networks.network_offering_id, networks.vpc_id,
> > networks.physical_network_id, networks.data_center_id, networks.related,
> > networks.guru_name, networks.state, networks.redundant, networks.dns1,
> > networks.domain_id, networks.account_id, networks.set_fields,
> > networks.guru_data, networks.dns2, networks.network_domain,
> > networks.removed, networks.created, networks.reservation_id,
> networks.uuid,
> > networks.guest_type, networks.acl_type, networks.restart_required,
> > networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr,
> > networks.display_network, networks.network_acl_id, networks.streched_l2,
> > networks.external_id FROM networks WHERE networks.id = ? AND
> > networks.removed IS NULL
> >
> > ### Getting Zone Information
> >
> > 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> > com.mysql.cj.jdbc.ClientPreparedStatement: SELECT data_center.id,
> > data_center.name, data_center.description, data_center.dns1,
> > data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2,
> > data_center.internal_dns1, data_center.internal_dns2,
> > data_center.router_mac_address, data_center.guest_network_cidr,
> > data_center.domain_id, data_center.domain, data_center.networktype,
> > data_center.dns_provider, data_center.dhcp_provider,
> > data_center.gateway_provider, data_center.vpn_provider,
> > data_center.userdata_provider, data_center.lb_provider,
> > data_center.firewall_provider, data_center.mac_address,
> > data_center.zone_token, data_center.removed,
> data_center.allocation_state,
> > data_center.uuid, data_center.is_security_group_enabled,
> > data_center.is_local_storage_enabled, data_center.sort_key FROM
> data_center
> > WHERE data_center.id = 1 AND data_center.removed IS NULL
> > 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> > consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> > 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> > connection
> > 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> > account_vlan_map.id, account_vlan_map.account_id,
> > account_vlan_map.vlan_db_id FROM account_vlan_map WHERE
> > account_vlan_map.account_id = ?
> >
> > ### Getting network information
> >
> > 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> > com.mysql.cj.jdbc.ClientPreparedStatement: SELECT networks.id,
> > networks.mode, networks.broadcast_domain_type, networks.traffic_type,
> > networks.name, networks.display_text, networks.broadcast_uri,
> > networks.gateway, networks.cidr, networks.network_cidr,
> > networks.network_offering_id, networks.vpc_id,
> > networks.physical_network_id, networks.data_center_id, networks.related,
> > networks.guru_name, networks.state, networks.redundant, networks.dns1,
> > networks.domain_id, networks.account_id, networks.set_fields,
> > networks.guru_data, networks.dns2, networks.network_domain,
> > networks.removed, networks.created, networks.reservation_id,
> networks.uuid,
> > networks.guest_type, networks.acl_type, networks.restart_required,
> > networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr,
> > networks.display_network, networks.network_acl_id, networks.streched_l2,
> > networks.external_id FROM networks WHERE networks.id = null AND
> > networks.removed IS NULL
> > 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> > consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> > 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> > connection
> > 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> > domain_vlan_map.id, domain_vlan_map.domain_id,
> domain_vlan_map.vlan_db_id
> > FROM domain_vlan_map WHERE domain_vlan_map.domain_id = ?
> >
> > #### Account_vlan_map - no entry for account "System" nor for Account
> > "Admin"
> >
> > 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> > com.mysql.cj.jdbc.ClientPreparedStatement: SELECT account_vlan_map.id,
> > account_vlan_map.account_id, account_vlan_map.vlan_db_id FROM
> > account_vlan_map WHERE account_vlan_map.account_id = 1
> > 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> > consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> > 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> > connection
> > 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT vlan.id
> ,
> > vlan.vlan_id, vlan.vlan_gateway, vlan.vlan_netmask, vlan.ip6_gateway,
> > vlan.ip6_cidr, vlan.data_center_id, vlan.ip4_range, vlan.ip6_range,
> > vlan.network_id, vlan.physical_network_id, vlan.vlan_type, vlan.uuid,
> > vlan.removed, vlan.created FROM vlan LEFT OUTER JOIN domain_vlan_map ON
> > vlan.id=domain_vlan_map.vlan_db_id LEFT OUTER JOIN account_vlan_map ON
> > vlan.id=account_vlan_map.vlan_db_id WHERE vlan.data_center_id = ? AND
> > vlan.removed IS NULL AND (domain_vlan_map.domain_id IS NULL ) AND
> > (account_vlan_map.account_id IS NULL )
> >
> > ### Checking for availeable VLANS wich are NOT assigned to ANY Account
> NOR
> > to ANY Domain --> Result is the intended vlan with expected IP Range
> >
> > 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> > com.mysql.cj.jdbc.ClientPreparedStatement: SELECT domain_vlan_map.id,
> > domain_vlan_map.domain_id, domain_vlan_map.vlan_db_id FROM
> domain_vlan_map
> > WHERE domain_vlan_map.domain_id = 1
> > 2022-08-31 18:44:21,214 TRACE [c.c.u.d.GenericDaoBase]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) join search statement is
> > com.mysql.cj.jdbc.ClientPreparedStatement: SELECT vlan.id, vlan.vlan_id,
> > vlan.vlan_gateway, vlan.vlan_netmask, vlan.ip6_gateway, vlan.ip6_cidr,
> > vlan.data_center_id, vlan.ip4_range, vlan.ip6_range, vlan.network_id,
> > vlan.physical_network_id, vlan.vlan_type, vlan.uuid, vlan.removed,
> > vlan.created FROM vlan LEFT OUTER JOIN domain_vlan_map ON vlan.id
> =domain_vlan_map.vlan_db_id
> > LEFT OUTER JOIN account_vlan_map ON vlan.id=account_vlan_map.vlan_db_id
> > WHERE vlan.data_center_id = 1 AND vlan.removed IS NULL AND
> > (domain_vlan_map.domain_id IS NULL ) AND (account_vlan_map.account_id IS
> > NULL )
> > 2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> > consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> > 2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> > connection
> > 2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> > configuration.instance, configuration.component, configuration.name,
> > configuration.value, configuration.default_value,
> > configuration.description, configuration.category,
> > configuration.is_dynamic, configuration.scope, configuration.updated FROM
> > configuration WHERE configuration.name = ?
> >
> > ### Getting further information on the vlan --> correct choice of
> expected
> > VLAN + IP-Range
> >
> > 2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> > com.mysql.cj.jdbc.ClientPreparedStatement: SELECT vlan.id, vlan.vlan_id,
> > vlan.vlan_gateway, vlan.vlan_netmask, vlan.ip6_gateway, vlan.ip6_cidr,
> > vlan.data_center_id, vlan.ip4_range, vlan.ip6_range, vlan.network_id,
> > vlan.physical_network_id, vlan.vlan_type, vlan.uuid, vlan.removed,
> > vlan.created FROM vlan LEFT OUTER JOIN domain_vlan_map ON vlan.id
> =domain_vlan_map.vlan_db_id
> > LEFT OUTER JOIN account_vlan_map ON vlan.id=account_vlan_map.vlan_db_id
> > WHERE vlan.data_center_id = 1 AND vlan.removed IS NULL AND
> > (domain_vlan_map.domain_id IS NULL ) AND (account_vlan_map.account_id IS
> > NULL )
> > 2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> > consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> > 2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> > connection
> > 2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> > user_ip_address.id, user_ip_address.account_id,
> > user_ip_address.domain_id, user_ip_address.public_ip_address,
> > user_ip_address.data_center_id, user_ip_address.source_nat,
> > user_ip_address.allocated, user_ip_address.vlan_db_id,
> > user_ip_address.one_to_one_nat, user_ip_address.vm_id,
> > user_ip_address.state, user_ip_address.mac_address,
> > user_ip_address.source_network_id, user_ip_address.network_id,
> > user_ip_address.uuid, user_ip_address.physical_network_id,
> > user_ip_address.is_system, user_ip_address.vpc_id,
> > user_ip_address.dnat_vmip, user_ip_address.is_portable,
> > user_ip_address.display, user_ip_address.rule_state,
> > user_ip_address.forsystemvms, user_ip_address.removed,
> > user_ip_address.created FROM user_ip_address INNER JOIN vlan ON
> > user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id
> =
> > ? AND user_ip_address.allocated IS NULL AND user_ip_address.vlan_db_id
> IN
> > (?,?,?,?) AND user_ip_address.forsystemvms = ? AND
> > user_ip_address.removed IS NULL AND (vlan.vlan_type = ? ) ORDER BY
> > user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id ASC LIMIT
> > 0, 1 FOR UPDATE
> >
> > ### Check of setting system.vm.public.ip.reservation.mode.strictness -->
> > value "true"
> >
> > 2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> > com.mysql.cj.jdbc.ClientPreparedStatement: SELECT configuration.instance,
> > configuration.component, configuration.name, configuration.value,
> > configuration.default_value, configuration.description,
> > configuration.category, configuration.is_dynamic, configuration.scope,
> > configuration.updated FROM configuration WHERE configuration.name =
> >
> x'73797374656D2E766D2E7075626C69632E69702E7265736572766174696F6E2E6D6F64652E7374726963746E657373'
> >
> > 2022-08-31 18:44:21,215 TRACE [c.c.u.d.GenericDaoBase]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) join search statement is
> > com.mysql.cj.jdbc.ClientPreparedStatement: SELECT user_ip_address.id,
> > user_ip_address.account_id, user_ip_address.domain_id,
> > user_ip_address.public_ip_address, user_ip_address.data_center_id,
> > user_ip_address.source_nat, user_ip_address.allocated,
> > user_ip_address.vlan_db_id, user_ip_address.one_to_one_nat,
> > user_ip_address.vm_id, user_ip_address.state,
> user_ip_address.mac_address,
> > user_ip_address.source_network_id, user_ip_address.network_id,
> > user_ip_address.uuid, user_ip_address.physical_network_id,
> > user_ip_address.is_system, user_ip_address.vpc_id,
> > user_ip_address.dnat_vmip, user_ip_address.is_portable,
> > user_ip_address.display, user_ip_address.rule_state,
> > user_ip_address.forsystemvms, user_ip_address.removed,
> > user_ip_address.created FROM user_ip_address INNER JOIN vlan ON
> > user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id
> =
> > 1 AND user_ip_address.allocated IS NULL AND user_ip_address.vlan_db_id
> IN
> > (3,10,13,14) AND user_ip_address.forsystemvms = 1 AND
> > user_ip_address.removed IS NULL AND (vlan.vlan_type = 'VirtualNetwork'
> )
> > ORDER BY user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id
> ASC
> > LIMIT 0, 1 FOR UPDATE
> > 2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> > connection
> > 2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> > account_details.id, account_details.account_id, account_details.name,
> > account_details.value FROM account_details WHERE
> account_details.account_id
> > = ? AND account_details.name = ? ORDER BY RAND() LIMIT 1
> >
> > ### Trying to fetch IP-Address: No value / ip address returned - Domain
> > "Root" shall provide ip addresses; thats where the vlanIDs are comming
> from
> > --> user_ip_address.vlan_db_id IN (3,10,13,14)
> >
> > 2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> > com.mysql.cj.jdbc.ClientPreparedStatement: SELECT user_ip_address.id,
> > user_ip_address.account_id, user_ip_address.domain_id,
> > user_ip_address.public_ip_address, user_ip_address.data_center_id,
> > user_ip_address.source_nat, user_ip_address.allocated,
> > user_ip_address.vlan_db_id, user_ip_address.one_to_one_nat,
> > user_ip_address.vm_id, user_ip_address.state,
> user_ip_address.mac_address,
> > user_ip_address.source_network_id, user_ip_address.network_id,
> > user_ip_address.uuid, user_ip_address.physical_network_id,
> > user_ip_address.is_system, user_ip_address.vpc_id,
> > user_ip_address.dnat_vmip, user_ip_address.is_portable,
> > user_ip_address.display, user_ip_address.rule_state,
> > user_ip_address.forsystemvms, user_ip_address.removed,
> > user_ip_address.created FROM user_ip_address INNER JOIN vlan ON
> > user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id
> =
> > 1 AND user_ip_address.allocated IS NULL AND user_ip_address.vlan_db_id
> IN
> > (3,10,13,14) AND user_ip_address.forsystemvms = 1 AND
> > user_ip_address.removed IS NULL AND (vlan.vlan_type = 'VirtualNetwork'
> )
> > ORDER BY user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id
> ASC
> > LIMIT 0, 1 FOR UPDATE
> > 2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> > consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> > 2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> > connection
> > 2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> > configuration.instance, configuration.component, configuration.name,
> > configuration.value, configuration.default_value,
> > configuration.description, configuration.category,
> > configuration.is_dynamic, configuration.scope, configuration.updated FROM
> > configuration WHERE configuration.name = ? ORDER BY RAND() LIMIT 1
> >
> > ### Trying to get information from the table account_detail for the
> > account "system". Table is empty so no result back
> >
> > 2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> > com.mysql.cj.jdbc.ClientPreparedStatement: SELECT account_details.id,
> > account_details.account_id, account_details.name, account_details.value
> > FROM account_details WHERE account_details.account_id = 1 AND
> > account_details.name = x'7573652E73797374656D2E7075626C69632E697073'
> > ORDER BY RAND() LIMIT 1
> > 2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> > consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> > 2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> > connection
> > 2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> > configuration.instance, configuration.component, configuration.name,
> > configuration.value, configuration.default_value,
> > configuration.description, configuration.category,
> > configuration.is_dynamic, configuration.scope, configuration.updated FROM
> > configuration WHERE configuration.name = ?
> >
> > ### Check for Configuration enable.account.settings.for.domain -->
> current
> > value = false
> >
> > 2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> > com.mysql.cj.jdbc.ClientPreparedStatement: SELECT configuration.instance,
> > configuration.component, configuration.name, configuration.value,
> > configuration.default_value, configuration.description,
> > configuration.category, configuration.is_dynamic, configuration.scope,
> > configuration.updated FROM configuration WHERE configuration.name =
> > x'656E61626C652E6163636F756E742E73657474696E67732E666F722E646F6D61696E'
> > ORDER BY RAND() LIMIT 1
> > 2022-08-31 18:44:21,217 WARN [c.c.n.IpAddressManagerImpl]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Unable to get ip address
> in
> > zone id=1, vlanId id=[3, 10, 13, 14]
> > 2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> > consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> > 2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> > connection
> > 2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> > data_center.id, data_center.name, data_center.description,
> > data_center.dns1, data_center.dns2, data_center.ip6_dns1,
> > data_center.ip6_dns2, data_center.internal_dns1,
> data_center.internal_dns2,
> > data_center.router_mac_address, data_center.guest_network_cidr,
> > data_center.domain_id, data_center.domain, data_center.networktype,
> > data_center.dns_provider, data_center.dhcp_provider,
> > data_center.gateway_provider, data_center.vpn_provider,
> > data_center.userdata_provider, data_center.lb_provider,
> > data_center.firewall_provider, data_center.mac_address,
> > data_center.zone_token, data_center.removed,
> data_center.allocation_state,
> > data_center.uuid, data_center.is_security_group_enabled,
> > data_center.is_local_storage_enabled, data_center.sort_key FROM
> data_center
> > WHERE data_center.id = ? AND data_center.removed IS NULL
> >
> > ### Check for setting use.system.public.ips --> current value = false
> >
> > 2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> > com.mysql.cj.jdbc.ClientPreparedStatement: SELECT configuration.instance,
> > configuration.component, configuration.name, configuration.value,
> > configuration.default_value, configuration.description,
> > configuration.category, configuration.is_dynamic, configuration.scope,
> > configuration.updated FROM configuration WHERE configuration.name =
> > x'7573652E73797374656D2E7075626C69632E697073'
> >
> > ### getting information for the current data_center / zone
> >
> > 2022-08-31 18:44:21,219 TRACE [c.c.u.d.T.Statement]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> > com.mysql.cj.jdbc.ClientPreparedStatement: SELECT data_center.id,
> > data_center.name, data_center.description, data_center.dns1,
> > data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2,
> > data_center.internal_dns1, data_center.internal_dns2,
> > data_center.router_mac_address, data_center.guest_network_cidr,
> > data_center.domain_id, data_center.domain, data_center.networktype,
> > data_center.dns_provider, data_center.dhcp_provider,
> > data_center.gateway_provider, data_center.vpn_provider,
> > data_center.userdata_provider, data_center.lb_provider,
> > data_center.firewall_provider, data_center.mac_address,
> > data_center.zone_token, data_center.removed,
> data_center.allocation_state,
> > data_center.uuid, data_center.is_security_group_enabled,
> > data_center.is_local_storage_enabled, data_center.sort_key FROM
> data_center
> > WHERE data_center.id = 1 AND data_center.removed IS NULL
> > 2022-08-31 18:44:21,219 DEBUG [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Rolling back the
> > transaction: Time = 17 Name = consoleproxy-1; called by
> >
> -TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-Transaction.execute:38-IpAddressManagerImpl.listAvailablePublicIps:824-IpAddressManagerImpl.fetchNewPublicIp:811-IpAddressManagerImpl.fetchNewPublicIp:803-IpAddressManagerImpl.assignPublicIpAddress:784-PublicNetworkGuru.getIp:125-PublicNetworkGuru.allocate:172-NetworkOrchestrator.allocateNic:973-NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice:823
> > 2022-08-31 18:44:21,219 TRACE [c.c.u.d.T.Lock]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) SQL
> > com.mysql.cj.jdbc.ClientPreparedStatement: SELECT user_ip_address.id,
> > user_ip_address.account_id, user_ip_address.domain_id,
> > user_ip_address.public_ip_address, user_ip_address.data_center_id,
> > user_ip_address.source_nat, user_ip_address.allocated,
> > user_ip_address.vlan_db_id, user_ip_address.one_to_one_nat,
> > user_ip_address.vm_id, user_ip_address.state,
> user_ip_address.mac_address,
> > user_ip_address.source_network_id, user_ip_address.network_id,
> > user_ip_address.uuid, user_ip_address.physical_network_id,
> > user_ip_address.is_system, user_ip_address.vpc_id,
> > user_ip_address.dnat_vmip, user_ip_address.is_portable,
> > user_ip_address.display, user_ip_address.rule_state,
> > user_ip_address.forsystemvms, user_ip_address.removed,
> > user_ip_address.created FROM user_ip_address INNER JOIN vlan ON
> > user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id
> =
> > 1 AND user_ip_address.allocated IS NULL AND user_ip_address.vlan_db_id
> IN
> > (3,10,13,14) AND user_ip_address.forsystemvms = 1 AND
> > user_ip_address.removed IS NULL AND (vlan.vlan_type = 'VirtualNetwork'
> )
> > ORDER BY user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id
> ASC
> > LIMIT 0, 1 FOR UPDATE took 4
> > 2022-08-31 18:44:21,219 TRACE [c.c.u.d.T.Connection]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing DB connection:
> > dbconn623913035
> > 2022-08-31 18:44:21,220 WARN [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Unable to allocate proxy
> > [Console VM instance {id: "86", name: "v-86-VM", uuid:
> > "6f77acb4-c65e-49e8-ae03-1175b3974412", type="ConsoleProxy"}] on zone [1]
> > due to [Insufficient address capacity].
> > com.cloud.exception.InsufficientAddressCapacityException: Insufficient
> > address capacityScope=interface com.cloud.dc.DataCenter; id=1
> >
> > following the settings of our global config which are mentioned in the
> > above log:
> > *system.vm.public.ip.reservation.mode.strictness* - true
> > *use.system.public.ips* - true
> > *enable.account.settings.for.domain* - false
> >
> > The challange now is to get all the things together into a context...
> > From my impression changeing the value for 'use.system.public.ips' may
> not
> > be helping, as at this point the "warning" already has been dropped. But
> > would only be a "workaround".
> >
> > Questions from my side:
> > - Looking into the web GUI for the IP-Ranges for the public network i am
> > getting quite curios. The current entry for '[Root] system' is strange.
> > Clicking on it i am getting a view stateing that this range shall be
> > "connected" to the Account 'System', the Domain 'Root' and SystemVMs
> 'true'.
> > While the 'System VMs = true' value can be found in the database, the
> vlan
> > / ip - range isn't connected to a account or domain at all in the
> database.
> > - What is the overall 'effect' of setting
> *system.vm.public.ip.reservation.mode.strictness?
> > *From the description for me a "true" will prevent the usage of of public
> > IPs from the system pool. But what is defined as "system Pool"? Looking
> at
> > the GUI I would assume that the "account" System has a dedicated pool of
> > public IPs for systemVMs (speaking that this would fail as there is no
> > reference in the DB for a mapping of any Pool / Range for the account
> > 'system').
> >
> > To summerize this seems more like a "unlucky" configuration or missing
> > db.entry / bug throughout versions. Or a mix of both.
> > How ever i don't really understand why there is a query to find VLANs not
> > attached to a Account or an Domain - or is this the SystemPool?
> >
> > Thanks again for all the help!
> > Chris
> >
> >
> >
> >
> >
> > Am Mi., 31. Aug. 2022 um 10:52 Uhr schrieb Slavka Peleva <
> > slavkap@storpool.com>:
> >
> >> Hi Chris,
> >>
> >> According to the logs you shared and looking at the code, I guess CS
> gets
> >> the VLAN IDs by domain id, not by zone id. But to be more precise in
> this,
> >> can you enable the TRACE level in your logs and share the exact query
> that
> >> CS executes?
> >> If I'm right with this, you can enable *just for now* the setting
> >> `use.system.public.ips` (To redeploy the system VMs). This should pick
> >> VLANs by zone ID. I hope that the deployment won't fail this time! If
> the
> >> system VMs are recreated, can you disable the setting
> >> `use.system.public.ips` (because I don't know if this will affect
> something
> >> else)?
> >>
> >> The discussion started about a problem migrating the system VMs, and
> >> probably many other issues will appear.
> >> Can you tell from which version you started the upgrade? (two weeks
> ago,
> >> I upgraded one of my environments from 4.11.3.0 to the main branch
> >> 4.18.0.0-SNAPSHOT without any issues)
> >> Did the upgrade was successful for the management/agents?
> >> Did you restart the agents on all machines? My guess is the new code
> >> isn't loaded. It's strange that CS is looking for systemvm.iso when 4.17
> >> doesn't know about it.
> >> Can you share the content in the directory
> >> `/usr/share/cloudstack-agent/lib/` from the hypervisor where the
> migration
> >> failed?
> >>
> >> Best regards,
> >> Slavka
> >>
> >>
> >>
> >> On Wed, Aug 31, 2022 at 2:35 AM Vash_X@gmx.de <Va...@gmx.de> wrote:
> >>
> >>> Hi Nicolas,
> >>>
> >>> > What is the value for the setting
> >>> ‘system.vm.public.ip.reservation.mode.strictness’?
> >>> This is set to 'true'.
> >>>
> >>> > Can you please check if this query gets any result?
> >>> Yes the query gives me results - all availeable "public" ip addresses
> of
> >>> the reserved range for the systemVMs
> >>>
> >>> # public_ip_address vlan_db_id vlan_id
> >>> 1 10.129.0.10 9 1000
> >>> 2 10.129.0.11 9 1000
> >>> 3 10.129.0.12 9 1000
> >>> 4 10.129.0.13 9 1000
> >>> 5 10.129.0.14 9 1000
> >>> 6 10.129.0.15 9 1000
> >>> 7 10.129.0.16 9 1000
> >>> 8 10.129.0.18 9 1000
> >>> 9 10.129.0.19 9 1000
> >>> Regarding the vlanId id=[3, 10, 13, 14]
> >>> I looked up my table "vlan" and am quite shocked:
> >>>
> >>> the ids 3, 13, 14 have a value for "range removed", so i don't know why
> >>> the
> >>> console proxy is looking for these networks.
> >>> the id "10" is is still active - however the ipv4 range is not "the
> right
> >>> one". That is assigned as the public ip-range for the root-account but
> >>> not
> >>> for use of systemvms...
> >>>
> >>> Am Di., 30. Aug. 2022 um 16:43 Uhr schrieb Nicolas Vazquez <
> >>> Nicolas.Vazquez@shapeblue.com>:
> >>>
> >>> > Hi Chris,
> >>> >
> >>> > What is the value for the setting
> >>> > ‘system.vm.public.ip.reservation.mode.strictness’?
> >>> >
> >>> > Looks like the error is not being able to get an IP address for zone
> >>> id=1,
> >>> > vlanId id=[3, 10, 13, 14].
> >>> > If I understood correctly you have defined a range for system VMs?
> Can
> >>> you
> >>> > please check if this query gets any result?
> >>> >
> >>> > select i.public_ip_address, i.vlan_db_id, v.vlan_id from
> >>> user_ip_address i
> >>> > join vlan v on (i.vlan_db_id = v.id)
> >>> > where i.forsystemvms = 1 and i.allocated is null and i.state = 'Free'
> >>> and
> >>> > v.removed is null and i.data_center_id = 1;
> >>> >
> >>> > Regards,
> >>> > Nicolas Vazquez
> >>> >
> >>> >
> >>> > From: Vash_X@gmx.de <Va...@gmx.de>
> >>> > Date: Tuesday, 30 August 2022 at 09:38
> >>> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>,
> Slavka
> >>> > Peleva <sl...@storpool.com>
> >>> > Subject: Re: Error migrate System VMs after upgrageing to 4.17
> >>> > Hi Slavka,
> >>> >
> >>> > that was my first thought, too. I checked beforehand and confirmed
> with
> >>> > your provided SQL Statements:
> >>> > *`select * from user_ip_address where forsystemvms = 1;`* <----
> >>> returns my
> >>> > defined (Public) IPs out of the correct range for SystemVMs
> >>> > *`SELECT * FROM clou*d.user_ip_address where allocated IS NULL AND
> >>> removed
> >>> > IS NULL; <-- All IPs shown beside the one which is currently in use
> >>> for the
> >>> > running ssvm. Meaning from my point of view "free" for allocation for
> >>> other
> >>> > system VMs.
> >>> >
> >>> > Querys performed directly on the DB-Server.
> >>> > I can't see a reason why there shouldn't be any IPs left for
> >>> allocation.
> >>> >
> >>> > Also it is interesting why the "reasoning" for the failing deployment
> >>> > changes from "no IP availeable" to "no useable volumes found".
> >>> > How ever i can't see anything missing at the moment, what makes me
> even
> >>> > more puzzeled.
> >>> >
> >>> > Regards
> >>> > Chris
> >>> >
> >>> > Am Di., 30. Aug. 2022 um 13:31 Uhr schrieb Slavka Peleva
> >>> > <sl...@storpool.com.invalid>:
> >>> >
> >>> > > Hi Chris,
> >>> > >
> >>> > > The deployment fails because CS cannot find a free IP address to
> >>> allocate
> >>> > > for the system VM. Can you check if you explicitly added a range
> for
> >>> the
> >>> > > system VMs, or you can check with this `select * from
> user_ip_address
> >>> > where
> >>> > > forsystemvms = 1;` If you don't have any ranges, you can get free
> IPs
> >>> > > with `SELECT
> >>> > > * FROM cloud.user_ip_address where allocated IS NULL AND removed
> IS
> >>> > NULL;`
> >>> > > . To get the exact query executed by CS, you need to increase the
> log
> >>> > level
> >>> > > to TRACE.
> >>> > >
> >>> > > Best regards,
> >>> > > Slavka
> >>> > >
> >>> >
> >>> >
> >>> >
> >>> > > On Mon, Aug 29, 2022 at 8:03 PM Vash_X@gmx.de <Va...@gmx.de>
> wrote:
> >>> > >
> >>> > > > Hi Slavka,
> >>> > > >
> >>> > > > no, i am not using any kind of custome offerings.
> >>> > > >
> >>> > > > I try to provide a more detailled log this time.
> >>> > > > Situation:
> >>> > > > v-83-VM was the "broken" consoleproxy. So the following log
> starts
> >>> with
> >>> > > > it's expunging.
> >>> > > > Afterwards the new consoleproxy shall be deployed - v-85-VM.
> >>> > > >
> >>> > > > This first deployment (logid:dcfe0999) fails due to Insufficent
> >>> address
> >>> > > > capacity - which i can't really understand or see in the
> database,
> >>> > > checking
> >>> > > > the availeable IPs for systemVMs (given that i might looking up
> the
> >>> > wrong
> >>> > > > tables....).
> >>> > > > How ever CS is working as expected trying to deploy the console
> >>> proxy
> >>> > > > (logid:440fd101) Sync job-4288).
> >>> > > > That job fails with "Unable to create deployment, no usable
> volumes
> >>> > found
> >>> > > > for the VM: 85".
> >>> > > > Afterwards i have another try to start the consoleproxy
> (job-4289)
> >>> > which
> >>> > > > fails for the same reason. And thats how things go on in the log
> >>> > file...
> >>> > > >
> >>> > > >
> >>> > > > 2022-08-29 17:01:16,224 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> >>> > (logid:5fe8559a)
> >>> > > > Cleaning up hypervisor data structures (ex. SRs in XenServer) for
> >>> > managed
> >>> > > > storage. Data from VM instance {id: "83", name: "v-83-VM", uuid:
> >>> > > > "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}.
> >>> > > > 2022-08-29 17:01:16,225 DEBUG [o.a.c.e.o.VolumeOrchestrator]
> >>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> >>> > (logid:5fe8559a)
> >>> > > > Cleaning storage for vm: 83
> >>> > > > 2022-08-29 17:01:16,233 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> >>> > (logid:5fe8559a)
> >>> > > > Expunged VM instance {id: "83", name: "v-83-VM", uuid:
> >>> > > > "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
> >>> > > > 2022-08-29 17:01:16,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> >>> > (logid:5fe8559a)
> >>> > > > Complete async job-4286, jobStatus: SUCCEEDED, resultCode: 0,
> >>> result:
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.api.response.SystemVmResponse/systemvm/{"id":"b1d1f9c6-eafe-4527-9335-e649c646aab0","systemvmtype":"consoleproxy","zoneid":"48e2e928-3300-43b5-8e3a-d9b358022502","zonename":"xxxxxxxxxx","dns1":"10.129.0.4","dns2":"195.10.195.195","name":"v-83-VM","templateid":"c20799e4-a990-449a-aae6-4b67374c7aff","templatename":"systemvm-kvm-4.17.0","created":"2022-08-29T10:38:26+0200","state":"Stopped","isdynamicallyscalable":"false","hasannotations":"false"}
> >>> > > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> >>> > (logid:5fe8559a)
> >>> > > > Publish async job-4286 complete on message bus
> >>> > > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> >>> > (logid:5fe8559a)
> >>> > > > Wake up jobs related to job-4286
> >>> > > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> >>> > (logid:5fe8559a)
> >>> > > > Update db status for job-4286
> >>> > > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> >>> > (logid:5fe8559a)
> >>> > > > Wake up jobs joined with job-4286 and disjoin all subjobs created
> >>> from
> >>> > > job-
> >>> > > > 4286
> >>> > > > 2022-08-29 17:01:16,260 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286) (logid:5fe8559a) Done
> >>> > > executing
> >>> > > >
> >>> org.apache.cloudstack.api.command.admin.systemvm.DestroySystemVmCmd for
> >>> > > > job-4286
> >>> > > > 2022-08-29 17:01:16,260 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286) (logid:5fe8559a)
> Remove
> >>> > > > job-4286 from job monitoring
> >>> > > > 2022-08-29 17:01:17,691 DEBUG [o.a.c.h.HAManagerImpl]
> >>> > > > (BackgroundTaskPollManager-6:ctx-6af18142) (logid:fe263a70) HA
> >>> health
> >>> > > check
> >>> > > > task is running...
> >>> > > >
> >>> > > > 2022-08-29 17:01:21,694 DEBUG [o.a.c.h.HAManagerImpl]
> >>> > > > (BackgroundTaskPollManager-6:ctx-89b1442f) (logid:0efc40b2) HA
> >>> health
> >>> > > check
> >>> > > > task is running...
> >>> > > > 2022-08-29 17:01:25,698 DEBUG [o.a.c.h.HAManagerImpl]
> >>> > > > (BackgroundTaskPollManager-5:ctx-62313d05) (logid:7073663f) HA
> >>> health
> >>> > > check
> >>> > > > task is running...
> >>> > > > 2022-08-29 17:01:25,767 DEBUG [c.c.s.StatsCollector]
> >>> > > > (secstorage-1:ctx-c80e5a6e) (logid:f8e3eb60) Verifying image
> >>> storage
> >>> > [1].
> >>> > > > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
> >>> > > > 2022-08-29 17:01:25,773 DEBUG
> [o.a.c.s.SecondaryStorageManagerImpl]
> >>> > > > (secstorage-1:ctx-c80e5a6e) (logid:f8e3eb60) Zone [1] is ready to
> >>> > launch
> >>> > > > secondary storage VM.
> >>> > > > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Zone 1 is ready to
> >>> > launch
> >>> > > > console proxy
> >>> > > > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Expand console
> proxy
> >>> > > standby
> >>> > > > capacity for zone xxxxxxxxxx
> >>> > > > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating console
> >>> proxy
> >>> > > > standby capacity for zone [1].
> >>> > > > 2022-08-29 17:01:26,032 INFO [c.c.c.ConsoleProxyManagerImpl]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) No stopped console
> >>> proxy
> >>> > > is
> >>> > > > available, need to allocate a new console proxy
> >>> > > > 2022-08-29 17:01:26,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Assign console
> proxy
> >>> > from
> >>> > > a
> >>> > > > newly started instance for request from data center : 1
> >>> > > > 2022-08-29 17:01:26,056 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing
> >>> network
> >>> > > > configuration for offering [Network Offering
> >>> > > > [1-Public-System-Public-Network]: Network {"id": 200, "name":
> >>> "null",
> >>> > > > "uuid": "1b0bf965-a391-478d-833d-3e37e4a15eb8",
> >>> "networkofferingid": 1}
> >>> > > > 2022-08-29 17:01:26,056 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
> >>> > > > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account
> >>> {"id": 1,
> >>> > > > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
> >>> > > > 2022-08-29 17:01:26,064 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing
> >>> network
> >>> > > > configuration for offering [Network Offering
> >>> > > > [3-Control-System-Control-Network]: Network {"id": 202, "name":
> >>> "null",
> >>> > > > "uuid": "2af42518-fbf4-4a35-9d6b-370e6ad315c7",
> >>> "networkofferingid": 3}
> >>> > > > 2022-08-29 17:01:26,064 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
> >>> > > > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account
> >>> {"id": 1,
> >>> > > > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
> >>> > > > 2022-08-29 17:01:26,069 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing
> >>> network
> >>> > > > configuration for offering [Network Offering
> >>> > > > [2-Management-System-Management-Network]: Network {"id": 201,
> >>> "name":
> >>> > > > "null", "uuid": "53711d87-a3c6-4f33-bfce-c30e0de72760",
> >>> > > > "networkofferingid": 2}
> >>> > > > 2022-08-29 17:01:26,069 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
> >>> > > > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account
> >>> {"id": 1,
> >>> > > > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
> >>> > > > 2022-08-29 17:01:26,077 INFO [c.c.v.VirtualMachineManagerImpl]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) allocating virtual
> >>> > machine
> >>> > > > from template:c20799e4-a990-449a-aae6-4b67374c7aff with
> >>> > hostname:v-85-VM
> >>> > > > and 3 networks
> >>> > > > 2022-08-29 17:01:26,079 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating entries
> >>> for
> >>> > VM:
> >>> > > > VM instance {id: "85", name: "v-85-VM", uuid:
> >>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
> >>> > > > 2022-08-29 17:01:26,082 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating nics
> for
> >>> VM
> >>> > > > instance {id: "85", name: "v-85-VM", uuid:
> >>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
> >>> > > > 2022-08-29 17:01:26,083 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating nic for
> >>> vm VM
> >>> > > > instance {id: "85", name: "v-85-VM", uuid:
> >>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} in
> >>> network
> >>> > > > Network {"id": 200, "name": "null", "uuid":
> >>> > > > "1b0bf965-a391-478d-833d-3e37e4a15eb8", "networkofferingid": 1}
> >>> with
> >>> > > > requested profile NicProfile
> >>> > > >
> >>> > >
> >>> >
> >>>
> {"broadcastUri":null,"iPv4Address":null,"id":0,"reservationId":null,"vmId":0}
> >>> > > > 2022-08-29 17:01:26,089 WARN [c.c.n.IpAddressManagerImpl]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to get ip
> >>> address
> >>> > > in
> >>> > > > zone id=1, vlanId id=[3, 10, 13, 14]
> >>> > > > 2022-08-29 17:01:26,090 DEBUG [c.c.u.d.T.Transaction]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Rolling back the
> >>> > > > transaction: Time = 8 Name = consoleproxy-1; called by
> >>> > > >
> >>> > >
> >>> >
> >>>
> -TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-Transaction.execute:38-IpAddressManagerImpl.listAvailablePublicIps:824-IpAddressManagerImpl.fetchNewPublicIp:811-IpAddressManagerImpl.fetchNewPublicIp:803-IpAddressManagerImpl.assignPublicIpAddress:784-PublicNetworkGuru.getIp:125-PublicNetworkGuru.allocate:172-NetworkOrchestrator.allocateNic:973-NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice:823
> >>> > > > 2022-08-29 17:01:26,090 WARN [c.c.c.ConsoleProxyManagerImpl]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to allocate
> >>> proxy
> >>> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> >>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
> >>> zone
> >>> > [1]
> >>> > > > due to [Insufficient address capacity].
> >>> > > > com.cloud.exception.InsufficientAddressCapacityException:
> >>> Insufficient
> >>> > > > address capacityScope=interface com.cloud.dc.DataCenter; id=1
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
> >>> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.network.IpAddressManagerImpl.listAvailablePublicIps(IpAddressManagerImpl.java:824)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl.java:811)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl.java:803)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.network.IpAddressManagerImpl.assignPublicIpAddress(IpAddressManagerImpl.java:784)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.network.guru.PublicNetworkGuru.getIp(PublicNetworkGuru.java:125)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.network.guru.PublicNetworkGuru.allocate(PublicNetworkGuru.java:172)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocateNic(NetworkOrchestrator.java:973)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice(NetworkOrchestrator.java:823)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$3.doInTransactionWithoutResult(NetworkOrchestrator.java:797)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
> >>> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocate(NetworkOrchestrator.java:778)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VirtualMachineManagerImpl$1.doInTransactionWithoutResult(VirtualMachineManagerImpl.java:478)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
> >>> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:469)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:528)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:700)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
> >>> > > > at
> >>> > >
> >>> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
> >>> > > > at
> >>> > > >
> >>> >
> >>>
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
> >>> > > > 2022-08-29 17:01:26,091 WARN [c.c.c.ConsoleProxyManagerImpl]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to allocate
> >>> > console
> >>> > > > proxy standby capacity for zone [1] due to [Unable to allocate
> >>> proxy
> >>> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> >>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
> >>> zone
> >>> > [1]
> >>> > > > due to [Insufficient address capacity].].
> >>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to
> allocate
> >>> > proxy
> >>> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> >>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
> >>> zone
> >>> > [1]
> >>> > > > due to [Insufficient address capacity].
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:704)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
> >>> > > > at
> >>> > >
> >>> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
> >>> > > > at
> >>> > > >
> >>> >
> >>>
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
> >>> > > > Caused by:
> >>> com.cloud.exception.InsufficientAddressCapacityException:
> >>> > > > Insufficient address capacityScope=interface
> >>> com.cloud.dc.DataCenter;
> >>> > > id=1
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
> >>> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> >>> > > > ... 38 more
> >>> > > > 2022-08-29 17:01:26,091 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) received console
> >>> proxy
> >>> > > alert
> >>> > > > 2022-08-29 17:01:26,092 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Console proxy
> >>> creation
> >>> > > > failure, zone: xxxxxxxxxx
> >>> > > > 2022-08-29 17:01:26,093 WARN [c.c.a.AlertManagerImpl]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) alertType=[10]
> >>> > > > dataCenterId=[1] podId=[null] clusterId=[null] message=[Console
> >>> proxy
> >>> > > > creation failure. zone: xxxxxxxxxx, error details: Unable to
> >>> allocate
> >>> > > proxy
> >>> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> >>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
> >>> zone
> >>> > [1]
> >>> > > > due to [Insufficient address capacity].].
> >>> > > > 2022-08-29 17:01:26,096 WARN [c.c.a.AlertManagerImpl]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) No recipients set
> in
> >>> > > global
> >>> > > > setting 'alert.email.addresses', skipping sending alert with
> >>> subject
> >>> > > > [Console proxy creation failure. zone: xxxxxxxxxx, error details:
> >>> > Unable
> >>> > > to
> >>> > > > allocate proxy [Console VM instance {id: "85", name: "v-85-VM",
> >>> uuid:
> >>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
> >>> zone
> >>> > [1]
> >>> > > > due to [Insufficient address capacity].] and content [Console
> proxy
> >>> > > > creation failure (zone xxxxxxxxxx)].
> >>> > > > 2022-08-29 17:01:26,096 WARN [c.c.v.SystemVmLoadScanner]
> >>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unexpected
> exception
> >>> > > Unable
> >>> > > > to allocate proxy [Console VM instance {id: "85", name:
> "v-85-VM",
> >>> > uuid:
> >>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
> >>> zone
> >>> > [1]
> >>> > > > due to [Insufficient address capacity].
> >>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to
> allocate
> >>> > proxy
> >>> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> >>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
> >>> zone
> >>> > [1]
> >>> > > > due to [Insufficient address capacity].
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:704)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
> >>> > > > at
> >>> > >
> >>> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
> >>> > > > at
> >>> > > >
> >>> >
> >>>
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
> >>> > > > Caused by:
> >>> com.cloud.exception.InsufficientAddressCapacityException:
> >>> > > > Insufficient address capacityScope=interface
> >>> com.cloud.dc.DataCenter;
> >>> > > id=1
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
> >>> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> >>> > > > ... 38 more
> >>> > > > 2022-08-29 17:01:29,700 DEBUG [o.a.c.h.HAManagerImpl]
> >>> > > > (BackgroundTaskPollManager-5:ctx-994465bd) (logid:1e2822fd) HA
> >>> health
> >>> > > check
> >>> > > > task is running...
> >>> > > > 2022-08-29 17:01:30,444 DEBUG
> >>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> >>> > > > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 2
> >>> routers
> >>> > to
> >>> > > > update status.
> >>> > > > 2022-08-29 17:01:30,449 DEBUG
> >>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> >>> > > > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 1
> >>> VPC's to
> >>> > > > update Redundant State.
> >>> > > > 2022-08-29 17:01:30,451 DEBUG
> >>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> >>> > > > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 0
> >>> networks
> >>> > to
> >>> > > > update RvR status.
> >>> > > > 2022-08-29 17:01:30,459 DEBUG [c.c.a.t.Request]
> >>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08)
> Seq
> >>> > > > 6-4833769775052102463: Sending { Cmd , MgmtId: 90520740759984,
> >>> via:
> >>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> >>> > > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
> >>> > > router.name
> >>> > >
> >>> >
> >>>
> ":"r-79-VM","router.ip":"169.254.144.198"},"wait":"30","bypassHostMaintenance":"false"}}]
> >>> > > > }
> >>> > > > 2022-08-29 17:01:30,643 DEBUG [c.c.a.t.Request]
> >>> > > > (AgentManager-Handler-12:null) (logid:) Seq
> 6-4833769775052102463:
> >>> > > > Processing: { Ans: , MgmtId: 90520740759984, via: 6, Ver: v1,
> >>> Flags:
> >>> > 10,
> >>> > > >
> >>> > >
> >>> >
> >>>
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"PRIMARY","result":"true","details":"Status:
> >>> > > > PRIMARY
> >>> > > > ","wait":"0","bypassHostMaintenance":"false"}}] }
> >>> > > > 2022-08-29 17:01:30,643 DEBUG [c.c.a.t.Request]
> >>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08)
> Seq
> >>> > > > 6-4833769775052102463: Received: { Ans: , MgmtId:
> 90520740759984,
> >>> via:
> >>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
> >>> CheckRouterAnswer } }
> >>> > > > 2022-08-29 17:01:30,643 DEBUG [c.c.a.m.AgentManagerImpl]
> >>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08)
> >>> Details
> >>> > > from
> >>> > > > executing class com.cloud.agent.api.CheckRouterCommand: Status:
> >>> PRIMARY
> >>> > > >
> >>> > > > 2022-08-29 17:01:30,655 DEBUG [c.c.a.t.Request]
> >>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08)
> Seq
> >>> > > > 1-5410793477308947401: Sending { Cmd , MgmtId: 90520740759984,
> >>> via:
> >>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> >>> > > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
> >>> > > router.name
> >>> > >
> >>> >
> >>>
> ":"r-80-VM","router.ip":"169.254.217.200"},"wait":"30","bypassHostMaintenance":"false"}}]
> >>> > > > }
> >>> > > > 2022-08-29 17:01:30,796 DEBUG [c.c.a.t.Request]
> >>> > > > (AgentManager-Handler-7:null) (logid:) Seq 1-5410793477308947401:
> >>> > > > Processing: { Ans: , MgmtId: 90520740759984, via: 1, Ver: v1,
> >>> Flags:
> >>> > 10,
> >>> > > >
> >>> > >
> >>> >
> >>>
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"BACKUP","result":"true","details":"Status:
> >>> > > > BACKUP
> >>> > > > ","wait":"0","bypassHostMaintenance":"false"}}] }
> >>> > > > 2022-08-29 17:01:30,796 DEBUG [c.c.a.t.Request]
> >>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08)
> Seq
> >>> > > > 1-5410793477308947401: Received: { Ans: , MgmtId:
> 90520740759984,
> >>> via:
> >>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
> >>> CheckRouterAnswer } }
> >>> > > > 2022-08-29 17:01:30,796 DEBUG [c.c.a.m.AgentManagerImpl]
> >>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08)
> >>> Details
> >>> > > from
> >>> > > > executing class com.cloud.agent.api.CheckRouterCommand: Status:
> >>> BACKUP
> >>> > > >
> >>> > > > 2022-08-29 17:01:33,703 DEBUG [o.a.c.h.HAManagerImpl]
> >>> > > > (BackgroundTaskPollManager-5:ctx-70bca202) (logid:b332dbe0) HA
> >>> health
> >>> > > check
> >>> > > > task is running...
> >>> > > > 2022-08-29 17:01:34,083 DEBUG [c.c.s.StatsCollector]
> >>> > > > (StatsCollector-6:ctx-213e2ecc) (logid:aa0584c9) AutoScaling
> >>> Monitor is
> >>> > > > running...
> >>> > > > 2022-08-29 17:01:37,707 DEBUG [o.a.c.h.HAManagerImpl]
> >>> > > > (BackgroundTaskPollManager-6:ctx-57493517) (logid:5093e75f) HA
> >>> health
> >>> > > check
> >>> > > > task is running...
> >>> > > > 2022-08-29 17:01:38,640 DEBUG [c.c.s.StatsCollector]
> >>> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c)
> >>> HostStatsCollector is
> >>> > > > running...
> >>> > > > 2022-08-29 17:01:38,655 DEBUG [c.c.a.t.Request]
> >>> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
> >>> > > 1-5410793477308947402:
> >>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
> >>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
> >>> GetHostStatsAnswer }
> >>> > }
> >>> > > > 2022-08-29 17:01:38,707 DEBUG [c.c.a.t.Request]
> >>> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
> >>> > > 5-8959067033723473395:
> >>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
> >>> > > > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
> >>> GetHostStatsAnswer }
> >>> > }
> >>> > > > 2022-08-29 17:01:38,717 DEBUG [c.c.a.t.Request]
> >>> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
> >>> > > 6-4833769775052102464:
> >>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
> >>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
> >>> GetHostStatsAnswer }
> >>> > }
> >>> > > > 2022-08-29 17:01:38,725 DEBUG [c.c.a.t.Request]
> >>> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
> >>> > > 7-295548725546192472:
> >>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
> >>> > > > 7(srv-5-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
> >>> GetHostStatsAnswer }
> >>> > }
> >>> > > > 2022-08-29 17:01:41,709 DEBUG [o.a.c.h.HAManagerImpl]
> >>> > > > (BackgroundTaskPollManager-5:ctx-a65ae462) (logid:bc2b7320) HA
> >>> health
> >>> > > check
> >>> > > > task is running...
> >>> > > > 2022-08-29 17:01:45,712 DEBUG [o.a.c.h.HAManagerImpl]
> >>> > > > (BackgroundTaskPollManager-2:ctx-05491ace) (logid:b2232c4d) HA
> >>> health
> >>> > > check
> >>> > > > task is running...
> >>> > > > 2022-08-29 17:01:47,019 DEBUG [c.c.a.m.AgentManagerImpl]
> >>> > > > (AgentManager-Handler-14:null) (logid:) Ping from Routing host
> >>> > > > 7(srv-5-comp.aaaaaaaa.bbb)
> >>> > > >
> >>> > > > 2022-08-29 17:01:49,715 DEBUG [o.a.c.h.HAManagerImpl]
> >>> > > > (BackgroundTaskPollManager-2:ctx-7943581c) (logid:504c98ff) HA
> >>> health
> >>> > > check
> >>> > > > task is running...
> >>> > > > 2022-08-29 17:01:51,881 DEBUG [c.c.a.m.AgentManagerImpl]
> >>> > > > (AgentManager-Handler-13:null) (logid:) Ping from Routing host
> >>> > > > 1(srv-2-comp.aaaaaaaa.bbb)
> >>> > > >
> >>> > > > 2022-08-29 17:01:53,718 DEBUG [o.a.c.h.HAManagerImpl]
> >>> > > > (BackgroundTaskPollManager-4:ctx-b11cfa22) (logid:3163613f) HA
> >>> health
> >>> > > check
> >>> > > > task is running...
> >>> > > > 2022-08-29 17:01:53,955 DEBUG [c.c.a.t.Request]
> >>> > > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
> >>> > > 1-5410793477308947403:
> >>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
> >>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
> GetVmStatsAnswer
> >>> } }
> >>> > > > 2022-08-29 17:01:54,025 DEBUG [c.c.a.t.Request]
> >>> > > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
> >>> > > 5-8959067033723473396:
> >>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
> >>> > > > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
> GetVmStatsAnswer
> >>> } }
> >>> > > > 2022-08-29 17:01:54,096 DEBUG [c.c.a.t.Request]
> >>> > > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
> >>> > > 6-4833769775052102465:
> >>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
> >>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
> GetVmStatsAnswer
> >>> } }
> >>> > > > 2022-08-29 17:01:54,217 DEBUG [c.c.a.t.Request]
> >>> > > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
> >>> > > 7-295548725546192473:
> >>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
> >>> > > > 7(srv-5-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
> GetVmStatsAnswer
> >>> } }
> >>> > > > 2022-08-29 17:01:55,765 DEBUG [c.c.s.StatsCollector]
> >>> > > > (secstorage-1:ctx-7d9995d7) (logid:e388a8b8) Verifying image
> >>> storage
> >>> > [1].
> >>> > > > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
> >>> > > > 2022-08-29 17:01:55,771 DEBUG
> [o.a.c.s.SecondaryStorageManagerImpl]
> >>> > > > (secstorage-1:ctx-7d9995d7) (logid:e388a8b8) Zone [1] is ready to
> >>> > launch
> >>> > > > secondary storage VM.
> >>> > > > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> >>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Zone 1 is ready to
> >>> > launch
> >>> > > > console proxy
> >>> > > > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> >>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Expand console
> proxy
> >>> > > standby
> >>> > > > capacity for zone xxxxxxxxxx
> >>> > > > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> >>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Allocating console
> >>> proxy
> >>> > > > standby capacity for zone [1].
> >>> > > > 2022-08-29 17:01:56,034 INFO [c.c.c.ConsoleProxyManagerImpl]
> >>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Found a stopped
> >>> console
> >>> > > > proxy, starting it. Vm id : 85
> >>> > > > 2022-08-29 17:01:56,035 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) start parameter
> >>> value of
> >>> > > > enterHardwareSetup == <very null> during processing of queued job
> >>> > > > 2022-08-29 17:01:56,044 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Sync job-4288
> >>> execution
> >>> > on
> >>> > > > object VmWorkJobQueue.85
> >>> > > > 2022-08-29 17:01:56,173 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (AsyncJobMgr-Heartbeat-1:ctx-e75826b1) (logid:e80fe659) Execute
> >>> > > sync-queue
> >>> > > > item: SyncQueueItemVO {id:2105, queueId: 2105, contentType:
> >>> AsyncJob,
> >>> > > > contentId: 4288, lastProcessMsid: 90520740759984,
> >>> lastprocessNumber: 1,
> >>> > > > lastProcessTime: Mon Aug 29 17:01:56 CEST 2022, created: Mon Aug
> 29
> >>> > > > 17:01:56 CEST 2022}
> >>> > > > 2022-08-29 17:01:56,174 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (AsyncJobMgr-Heartbeat-1:ctx-e75826b1) (logid:e80fe659) Schedule
> >>> queued
> >>> > > > job-4288
> >>> > > > 2022-08-29 17:01:56,179 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
> >>> (logid:7e671dd0)
> >>> > > Add
> >>> > > > job-4288 into job monitoring
> >>> > > > 2022-08-29 17:01:56,186 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
> >>> (logid:ba10fec5)
> >>> > > > Executing AsyncJobVO: {id:4288, userId: 1, accountId: 1,
> >>> instanceType:
> >>> > > > null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
> >>> > > >
> >>> > >
> >>> >
> >>>
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAVXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
> >>> > > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
> >>> 0,
> >>> > > > result: null, initMsid: 90520740759984, completeMsid: null,
> >>> > lastUpdated:
> >>> > > > null, lastPolled: null, created: Mon Aug 29 17:01:56 CEST 2022,
> >>> > removed:
> >>> > > > null}
> >>> > > > 2022-08-29 17:01:56,186 DEBUG [c.c.v.VmWorkJobDispatcher]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
> >>> (logid:ba10fec5)
> >>> > > Run
> >>> > > > VM work job: com.cloud.vm.VmWorkStart for VM 85, job origin: 3445
> >>> > > > 2022-08-29 17:01:56,187 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a)
> >>> > > > (logid:ba10fec5) Execute VM work job:
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> >>> > > > 2022-08-29 17:01:56,189 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a)
> >>> > > > (logid:ba10fec5) orchestrating VM start for 'v-85-VM'
> >>> > > > com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
> >>> > > > 2022-08-29 17:01:56,196 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a)
> >>> > > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> >>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
> state
> >>> > > > transited from [Stopped] to [Starting] with event
> [StartRequested].
> >>> > VM's
> >>> > > > original host: null, new host: null, host before state
> transition:
> >>> null
> >>> > > > 2022-08-29 17:01:56,196 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a)
> >>> > > > (logid:ba10fec5) Successfully transitioned to start state for VM
> >>> > instance
> >>> > > > {id: "85", name: "v-85-VM", uuid:
> >>> > "110d8620-c029-4ca9-9084-210099725a80",
> >>> > > > type="ConsoleProxy"} reservation id =
> >>> > > 0ab68202-d8be-469c-9b2a-87a55c2dc489
> >>> > > > 2022-08-29 17:01:56,200 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a)
> >>> > > > (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts:
> >>> null
> >>> > > > 2022-08-29 17:01:56,202 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a)
> >>> > > > (logid:ba10fec5) VM start attempt #1
> >>> > > > 2022-08-29 17:01:56,203 DEBUG
> [c.c.d.DeploymentPlanningManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a)
> >>> > > > (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
> >>> > > > 2022-08-29 17:01:56,203 DEBUG
> [c.c.d.DeploymentPlanningManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a)
> >>> > > > (logid:ba10fec5) Trying to allocate a host and storage pools from
> >>> dc:1,
> >>> > > > pod:null,cluster:null, requested cpu: 500, requested ram: (1.00
> GB)
> >>> > > > 1073741824
> >>> > > > 2022-08-29 17:01:56,203 DEBUG
> [c.c.d.DeploymentPlanningManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a)
> >>> > > > (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?:
> No
> >>> > > > 2022-08-29 17:01:56,212 DEBUG
> [c.c.d.DeploymentPlanningManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a)
> >>> > > > (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts:
> null
> >>> > > > 2022-08-29 17:01:56,214 DEBUG [c.c.d.FirstFitPlanner]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a)
> >>> > > > (logid:ba10fec5) Searching all possible resources under this
> Zone:
> >>> 1
> >>> > > > 2022-08-29 17:01:56,215 DEBUG [c.c.d.FirstFitPlanner]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a)
> >>> > > > (logid:ba10fec5) Listing clusters in order of aggregate capacity,
> >>> that
> >>> > > have
> >>> > > > (at least one host with) enough CPU and RAM capacity under this
> >>> Zone: 1
> >>> > > > 2022-08-29 17:01:56,218 DEBUG [c.c.d.FirstFitPlanner]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a)
> >>> > > > (logid:ba10fec5) Removing from the clusterId list these clusters
> >>> from
> >>> > > avoid
> >>> > > > set: []
> >>> > > > 2022-08-29 17:01:56,224 DEBUG
> [c.c.d.DeploymentPlanningManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a)
> >>> > > > (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
> >>> > > > 2022-08-29 17:01:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested
> >>> with
> >>> > > > Custom[UEFI] Boot Type false
> >>> > > > 2022-08-29 17:01:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in
> >>> dc: 1
> >>> > > > pod:1 cluster:1
> >>> > > > 2022-08-29 17:01:56,226 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has
> 4
> >>> > hosts
> >>> > > to
> >>> > > > check for allocation: [Host {"id": "5", "name":
> >>> > > "srv-4-comp.aaaaaaaa.bbb",
> >>> > > > "uuid": "134c625d-15ab-461a-a38d-36fb0b979b5d",
> "type"="Routing"},
> >>> Host
> >>> > > > {"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb", "uuid":
> >>> > > > "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
> >>> {"id":
> >>> > > "6",
> >>> > > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> >>> > > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
> >>> {"id":
> >>> > > "1",
> >>> > > > "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> >>> > > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> >>> > > > 2022-08-29 17:01:56,232 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for
> >>> allocation
> >>> > > > after prioritization: [Host {"id": "5", "name":
> >>> > > "srv-4-comp.aaaaaaaa.bbb",
> >>> > > > "uuid": "134c625d-15ab-461a-a38d-36fb0b979b5d",
> "type"="Routing"},
> >>> Host
> >>> > > > {"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb", "uuid":
> >>> > > > "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
> >>> {"id":
> >>> > > "6",
> >>> > > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> >>> > > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
> >>> {"id":
> >>> > > "1",
> >>> > > > "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> >>> > > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> >>> > > > 2022-08-29 17:01:56,232 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for
> >>> speed=500Mhz,
> >>> > > > Ram=1024 MB
> >>> > > > 2022-08-29 17:01:56,232 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
> >>> > > > srv-4-comp.aaaaaaaa.bbb, uuid:
> >>> 134c625d-15ab-461a-a38d-36fb0b979b5d} is
> >>> > > KVM
> >>> > > > hypervisor type, no max guest limit check needed
> >>> > > > 2022-08-29 17:01:56,234 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu
> >>> capability
> >>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
> >>> speed:
> >>> > 500
> >>> > > > 2022-08-29 17:01:56,234 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5
> has
> >>> > enough
> >>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> >>> > 1073741824 ,
> >>> > > > cpuOverprovisioningFactor: 1.0
> >>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
> >>> CPU:
> >>> > > 127200
> >>> > > > and CPU after applying overprovisioning: 127200
> >>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 ,
> >>> Requested
> >>> > > > CPU: 500
> >>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
> >>> > > > 251009040384 , Requested RAM: (1.00 GB) 1073741824
> >>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
> and
> >>> RAM
> >>> > > > available
> >>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
> >>> from
> >>> > > host:
> >>> > > > 5, used: 11100, reserved: 0, actual total: 127200, total with
> >>> > > > overprovisioning: 127200; requested
> >>> cpu:500,alloc_from_last_host?:false
> >>> > > > ,considerReservedCapacity?: true
> >>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
> >>> from
> >>> > > host:
> >>> > > > 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total:
> >>> (250.27
> >>> > > GB)
> >>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
> >>> > alloc_from_last_host?:
> >>> > > > false , considerReservedCapacity?: true
> >>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> >>> > adding
> >>> > > to
> >>> > > > list: 5
> >>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
> >>> > > > srv-5-comp.aaaaaaaa.bbb, uuid:
> >>> 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is
> >>> > > KVM
> >>> > > > hypervisor type, no max guest limit check needed
> >>> > > > 2022-08-29 17:01:56,238 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu
> >>> capability
> >>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
> >>> speed:
> >>> > 500
> >>> > > > 2022-08-29 17:01:56,238 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7
> has
> >>> > enough
> >>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> >>> > 1073741824 ,
> >>> > > > cpuOverprovisioningFactor: 1.0
> >>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
> >>> CPU:
> >>> > > 127200
> >>> > > > and CPU after applying overprovisioning: 127200
> >>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 ,
> >>> Requested
> >>> > > CPU:
> >>> > > > 500
> >>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
> >>> > > > 189373743104 , Requested RAM: (1.00 GB) 1073741824
> >>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
> and
> >>> RAM
> >>> > > > available
> >>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
> >>> from
> >>> > > host:
> >>> > > > 7, used: 90100, reserved: 0, actual total: 127200, total with
> >>> > > > overprovisioning: 127200; requested
> >>> cpu:500,alloc_from_last_host?:false
> >>> > > > ,considerReservedCapacity?: true
> >>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
> >>> from
> >>> > > host:
> >>> > > > 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total:
> >>> (250.27
> >>> > > GB)
> >>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
> >>> > alloc_from_last_host?:
> >>> > > > false , considerReservedCapacity?: true
> >>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> >>> > adding
> >>> > > to
> >>> > > > list: 7
> >>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
> >>> > > > srv-3-comp.aaaaaaaa.bbb, uuid:
> >>> 6858713b-7e17-46d8-854d-24a13f9c7001} is
> >>> > > KVM
> >>> > > > hypervisor type, no max guest limit check needed
> >>> > > > 2022-08-29 17:01:56,242 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu
> >>> capability
> >>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
> >>> speed:
> >>> > 500
> >>> > > > 2022-08-29 17:01:56,242 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6
> has
> >>> > enough
> >>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> >>> > 1073741824 ,
> >>> > > > cpuOverprovisioningFactor: 1.0
> >>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
> >>> CPU:
> >>> > > 127200
> >>> > > > and CPU after applying overprovisioning: 127200
> >>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 ,
> >>> Requested
> >>> > > CPU:
> >>> > > > 500
> >>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
> >>> > > > 204032835584 , Requested RAM: (1.00 GB) 1073741824
> >>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
> and
> >>> RAM
> >>> > > > available
> >>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
> >>> from
> >>> > > host:
> >>> > > > 6, used: 42900, reserved: 0, actual total: 127200, total with
> >>> > > > overprovisioning: 127200; requested
> >>> cpu:500,alloc_from_last_host?:false
> >>> > > > ,considerReservedCapacity?: true
> >>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
> >>> from
> >>> > > host:
> >>> > > > 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total:
> >>> (250.27
> >>> > > GB)
> >>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
> >>> > alloc_from_last_host?:
> >>> > > > false , considerReservedCapacity?: true
> >>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> >>> > adding
> >>> > > to
> >>> > > > list: 6
> >>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
> >>> > > > srv-2-comp.aaaaaaaa.bbb, uuid:
> >>> 883f7f62-55f2-42f4-8242-16b6847c4204} is
> >>> > > KVM
> >>> > > > hypervisor type, no max guest limit check needed
> >>> > > > 2022-08-29 17:01:56,246 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu
> >>> capability
> >>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
> >>> speed:
> >>> > 500
> >>> > > > 2022-08-29 17:01:56,246 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1
> has
> >>> > enough
> >>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> >>> > 1073741824 ,
> >>> > > > cpuOverprovisioningFactor: 1.0
> >>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
> >>> CPU:
> >>> > > 127200
> >>> > > > and CPU after applying overprovisioning: 127200
> >>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 ,
> >>> Requested
> >>> > > CPU:
> >>> > > > 500
> >>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
> >>> > > > 229802639360 , Requested RAM: (1.00 GB) 1073741824
> >>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
> and
> >>> RAM
> >>> > > > available
> >>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
> >>> from
> >>> > > host:
> >>> > > > 1, used: 48200, reserved: 0, actual total: 127200, total with
> >>> > > > overprovisioning: 127200; requested
> >>> cpu:500,alloc_from_last_host?:false
> >>> > > > ,considerReservedCapacity?: true
> >>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
> >>> from
> >>> > > host:
> >>> > > > 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total:
> >>> (250.27
> >>> > > GB)
> >>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
> >>> > alloc_from_last_host?:
> >>> > > > false , considerReservedCapacity?: true
> >>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> >>> > adding
> >>> > > to
> >>> > > > list: 1
> >>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator
> >>> returning 4
> >>> > > > suitable hosts
> >>> > > > 2022-08-29 17:01:56,258 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a)
> >>> > > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> >>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
> state
> >>> > > > transited from [Starting] to [Stopped] with event
> >>> [OperationFailed].
> >>> > VM's
> >>> > > > original host: null, new host: null, host before state
> transition:
> >>> null
> >>> > > > 2022-08-29 17:01:56,258 WARN [c.c.v.VirtualMachineManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a)
> >>> > > > (logid:ba10fec5) Unable to orchestrate start VM instance {id:
> "85",
> >>> > name:
> >>> > > > "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
> >>> > > > type="ConsoleProxy"} due to [Unable to create deployment, no
> usable
> >>> > > volumes
> >>> > > > found for the VM: 85].
> >>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to create
> >>> > > > deployment, no usable volumes found for the VM: 85
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
> >>> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> >>> > Source)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> >>> > > > at
> >>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> >>> > > > at
> >>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
> >>> > > > 2022-08-29 17:01:56,259 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a)
> >>> > > > (logid:ba10fec5) Done executing VM work job:
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> >>> > > > 2022-08-29 17:01:56,260 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a)
> >>> > > > (logid:ba10fec5) Complete async job-4288, jobStatus: FAILED,
> >>> > resultCode:
> >>> > > 0,
> >>> > > > result:
> >>> > > >
> >>> > >
> >>> >
> >>>
> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODUiLCBuYW1lOiAidi04NS1WTSIsIHV1aWQ6ICIxMTBkODYyMC1jMDI5LTRjYTktOTA4NC0yMTAwOTk3MjVhODAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODVdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
> >>> > > > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a)
> >>> > > > (logid:ba10fec5) Publish async job-4288 complete on message bus
> >>> > > > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a)
> >>> > > > (logid:ba10fec5) Wake up jobs related to job-4288
> >>> > > > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a)
> >>> > > > (logid:ba10fec5) Update db status for job-4288
> >>> > > > 2022-08-29 17:01:56,262 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288
> ctx-2852a95a)
> >>> > > > (logid:ba10fec5) Wake up jobs joined with job-4288 and disjoin
> all
> >>> > > subjobs
> >>> > > > created from job- 4288
> >>> > > > 2022-08-29 17:01:56,270 DEBUG [c.c.v.VmWorkJobDispatcher]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
> >>> (logid:ba10fec5)
> >>> > > > Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 85,
> >>> job
> >>> > > > origin: 3445
> >>> > > > 2022-08-29 17:01:56,270 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
> >>> (logid:ba10fec5)
> >>> > > > Done executing com.cloud.vm.VmWorkStart for job-4288
> >>> > > > 2022-08-29 17:01:56,272 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
> >>> (logid:ba10fec5)
> >>> > > > Remove job-4288 from job monitoring
> >>> > > > 2022-08-29 17:01:56,277 WARN [c.c.c.ConsoleProxyManagerImpl]
> >>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Unable to allocate
> >>> > console
> >>> > > > proxy standby capacity for zone [1] due to [Unable to orchestrate
> >>> start
> >>> > > VM
> >>> > > > instance {id: "85", name: "v-85-VM", uuid:
> >>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due
> to
> >>> > > [Unable
> >>> > > > to create deployment, no usable volumes found for the VM: 85].].
> >>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to
> >>> orchestrate
> >>> > > > start VM instance {id: "85", name: "v-85-VM", uuid:
> >>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due
> to
> >>> > > [Unable
> >>> > > > to create deployment, no usable volumes found for the VM: 85].
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> >>> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> >>> > Source)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> >>> > > > at
> >>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> >>> > > > at
> >>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
> >>> > > > 2022-08-29 17:01:56,277 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> >>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) received console
> >>> proxy
> >>> > > alert
> >>> > > > 2022-08-29 17:01:56,278 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> >>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Console proxy
> >>> creation
> >>> > > > failure, zone: xxxxxxxxxx
> >>> > > > 2022-08-29 17:01:56,279 WARN [c.c.a.AlertManagerImpl]
> >>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) alertType=[10]
> >>> > > > dataCenterId=[1] podId=[null] clusterId=[null] message=[Console
> >>> proxy
> >>> > > > creation failure. zone: xxxxxxxxxx, error details: Unable to
> >>> > orchestrate
> >>> > > > start VM instance {id: "85", name: "v-85-VM", uuid:
> >>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due
> to
> >>> > > [Unable
> >>> > > > to create deployment, no usable volumes found for the VM: 85].].
> >>> > > > 2022-08-29 17:01:56,284 WARN [c.c.a.AlertManagerImpl]
> >>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) No recipients set
> in
> >>> > > global
> >>> > > > setting 'alert.email.addresses', skipping sending alert with
> >>> subject
> >>> > > > [Console proxy creation failure. zone: xxxxxxxxxx, error details:
> >>> > Unable
> >>> > > to
> >>> > > > orchestrate start VM instance {id: "85", name: "v-85-VM", uuid:
> >>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due
> to
> >>> > > [Unable
> >>> > > > to create deployment, no usable volumes found for the VM: 85].]
> and
> >>> > > content
> >>> > > > [Console proxy creation failure (zone xxxxxxxxxx)].
> >>> > > > 2022-08-29 17:01:56,284 WARN [c.c.v.SystemVmLoadScanner]
> >>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Unexpected
> exception
> >>> > > Unable
> >>> > > > to orchestrate start VM instance {id: "85", name: "v-85-VM",
> uuid:
> >>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due
> to
> >>> > > [Unable
> >>> > > > to create deployment, no usable volumes found for the VM: 85].
> >>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to
> >>> orchestrate
> >>> > > > start VM instance {id: "85", name: "v-85-VM", uuid:
> >>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due
> to
> >>> > > [Unable
> >>> > > > to create deployment, no usable volumes found for the VM: 85].
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> >>> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> >>> > Source)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> >>> > > > at
> >>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> >>> > > > at
> >>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
> >>> > > > 2022-08-29 17:01:57,721 DEBUG [o.a.c.h.HAManagerImpl]
> >>> > > > (BackgroundTaskPollManager-4:ctx-9528226f) (logid:4512ff9b) HA
> >>> health
> >>> > > check
> >>> > > > task is running...
> >>> > > > 2022-08-29 17:01:59,504 DEBUG [c.c.s.StatsCollector]
> >>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) StorageCollector
> >>> is
> >>> > > > running...
> >>> > > > 2022-08-29 17:01:59,518 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> >>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
> >>> > > getCommandHostDelegation:
> >>> > > > class com.cloud.agent.api.GetStorageStatsCommand
> >>> > > > 2022-08-29 17:01:59,518 DEBUG [c.c.h.XenServerGuru]
> >>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning
> >>> the
> >>> > > > default host to execute commands because the command is not of
> Copy
> >>> > type.
> >>> > > > 2022-08-29 17:01:59,571 DEBUG [c.c.a.t.Request]
> >>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
> >>> > > > 11-7523826127475836583: Received: { Ans: , MgmtId:
> 90520740759984,
> >>> > via:
> >>> > > > 11(s-32-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> >>> > > > 2022-08-29 17:01:59,574 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> >>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
> >>> > > getCommandHostDelegation:
> >>> > > > class com.cloud.agent.api.GetStorageStatsCommand
> >>> > > > 2022-08-29 17:01:59,574 DEBUG [c.c.h.XenServerGuru]
> >>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning
> >>> the
> >>> > > > default host to execute commands because the command is not of
> Copy
> >>> > type.
> >>> > > > 2022-08-29 17:01:59,615 DEBUG [c.c.a.t.Request]
> >>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
> >>> > > 5-8959067033723473397:
> >>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
> >>> > > > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
> >>> GetStorageStatsAnswer
> >>> > > } }
> >>> > > > 2022-08-29 17:01:59,624 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> >>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
> >>> > > getCommandHostDelegation:
> >>> > > > class com.cloud.agent.api.GetStorageStatsCommand
> >>> > > > 2022-08-29 17:01:59,624 DEBUG [c.c.h.XenServerGuru]
> >>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning
> >>> the
> >>> > > > default host to execute commands because the command is not of
> Copy
> >>> > type.
> >>> > > > 2022-08-29 17:01:59,653 DEBUG [c.c.a.t.Request]
> >>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
> >>> > > 6-4833769775052102466:
> >>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
> >>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
> >>> GetStorageStatsAnswer
> >>> > > } }
> >>> > > > 2022-08-29 17:01:59,979 DEBUG [c.c.a.m.AgentManagerImpl]
> >>> > > > (AgentManager-Handler-12:null) (logid:) Ping from Routing host
> >>> > > > 5(srv-4-comp.aaaaaaaa.bbb)
> >>> > > >
> >>> > > > 2022-08-29 17:02:00,269 DEBUG [c.c.s.StatsCollector]
> >>> > > > (StatsCollector-5:ctx-d2b6ffd3) (logid:95201678) DbCollector is
> >>> > > running...
> >>> > > > 2022-08-29 17:02:00,443 DEBUG
> >>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> >>> > > > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 2
> >>> routers
> >>> > to
> >>> > > > update status.
> >>> > > > 2022-08-29 17:02:00,449 DEBUG
> >>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> >>> > > > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 1
> >>> VPC's to
> >>> > > > update Redundant State.
> >>> > > > 2022-08-29 17:02:00,450 DEBUG
> >>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> >>> > > > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 0
> >>> networks
> >>> > to
> >>> > > > update RvR status.
> >>> > > > 2022-08-29 17:02:00,457 DEBUG [c.c.a.t.Request]
> >>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f)
> Seq
> >>> > > > 6-4833769775052102467: Sending { Cmd , MgmtId: 90520740759984,
> >>> via:
> >>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> >>> > > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
> >>> > > router.name
> >>> > >
> >>> >
> >>>
> ":"r-79-VM","router.ip":"169.254.144.198"},"wait":"30","bypassHostMaintenance":"false"}}]
> >>> > > > }
> >>> > > > 2022-08-29 17:02:00,599 DEBUG [c.c.a.t.Request]
> >>> > > > (AgentManager-Handler-7:null) (logid:) Seq 6-4833769775052102467:
> >>> > > > Processing: { Ans: , MgmtId: 90520740759984, via: 6, Ver: v1,
> >>> Flags:
> >>> > 10,
> >>> > > >
> >>> > >
> >>> >
> >>>
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"PRIMARY","result":"true","details":"Status:
> >>> > > > PRIMARY
> >>> > > > ","wait":"0","bypassHostMaintenance":"false"}}] }
> >>> > > > 2022-08-29 17:02:00,599 DEBUG [c.c.a.t.Request]
> >>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f)
> Seq
> >>> > > > 6-4833769775052102467: Received: { Ans: , MgmtId:
> 90520740759984,
> >>> via:
> >>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
> >>> CheckRouterAnswer } }
> >>> > > > 2022-08-29 17:02:00,599 DEBUG [c.c.a.m.AgentManagerImpl]
> >>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f)
> >>> Details
> >>> > > > from executing class com.cloud.agent.api.CheckRouterCommand:
> >>> Status:
> >>> > > PRIMARY
> >>> > > >
> >>> > > > 2022-08-29 17:02:00,612 DEBUG [c.c.a.t.Request]
> >>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f)
> Seq
> >>> > > > 1-5410793477308947404: Sending { Cmd , MgmtId: 90520740759984,
> >>> via:
> >>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> >>> > > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
> >>> > > router.name
> >>> > >
> >>> >
> >>>
> ":"r-80-VM","router.ip":"169.254.217.200"},"wait":"30","bypassHostMaintenance":"false"}}]
> >>> > > > }
> >>> > > > 2022-08-29 17:02:00,748 DEBUG [c.c.a.m.AgentManagerImpl]
> >>> > > > (AgentManager-Handler-4:null) (logid:) Ping from 11(s-32-VM)
> >>> > > > 2022-08-29 17:02:00,750 DEBUG [c.c.a.t.Request]
> >>> > > > (AgentManager-Handler-1:null) (logid:) Seq 1-5410793477308947404:
> >>> > > > Processing: { Ans: , MgmtId: 90520740759984, via: 1, Ver: v1,
> >>> Flags:
> >>> > 10,
> >>> > > >
> >>> > >
> >>> >
> >>>
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"BACKUP","result":"true","details":"Status:
> >>> > > > BACKUP
> >>> > > > ","wait":"0","bypassHostMaintenance":"false"}}] }
> >>> > > > 2022-08-29 17:02:00,750 DEBUG [c.c.a.t.Request]
> >>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f)
> Seq
> >>> > > > 1-5410793477308947404: Received: { Ans: , MgmtId:
> 90520740759984,
> >>> via:
> >>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
> >>> CheckRouterAnswer } }
> >>> > > > 2022-08-29 17:02:00,751 DEBUG [c.c.a.m.AgentManagerImpl]
> >>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f)
> >>> Details
> >>> > > > from executing class com.cloud.agent.api.CheckRouterCommand:
> >>> Status:
> >>> > > BACKUP
> >>> > > >
> >>> > > > 2022-08-29 17:02:01,362 DEBUG [c.c.s.StatsCollector]
> >>> > > > (Cluster-Worker-839:ctx-e097a8ab) (logid:3deacf65) StatusUpdate
> >>> from
> >>> > > > 90520740759984, json:
> >>> > > >
> >>> > >
> >>> >
> >>>
> {"managementServerHostId":1,"managementServerHostUuid":"5615ca5f-0716-4da1-93ec-31f79fd19bbe","collectionTime":"Aug
> >>> > > > 29, 2022, 5:02:01
> >>> > > >
> >>> > >
> >>> >
> >>>
> PM","sessions":1,"cpuUtilization":0.0,"totalJvmMemoryBytes":324534272,"freeJvmMemoryBytes":68595424,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":103760388,"jvmStartTime":1661681560910,"availableProcessors":6,"loadAverage":0.02,"totalInit":139788288,"totalUsed":474791104,"totalCommitted":557662208,"pid":1522332,"jvmName":"1522332@vm-1-cs-mgmt.aaaaaaaa.bbb
> >>> > >
> >>> >
> >>>
> ","jvmVendor":"Ubuntu","jvmVersion":"11.0.16+8-post-Ubuntu-0ubuntu120.04","osDistribution":"Ubuntu
> >>> > > > 20.04.5
> >>> > > >
> >>> > >
> >>> >
> >>>
> LTS","agentCount":5,"heapMemoryUsed":256775312,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":13,"threadsRunnableCount":15,"threadsTerminatedCount":0,"threadsTotalCount":420,"threadsWaitingCount":331,"systemMemoryTotal":8343236608,"systemMemoryFree":2623193088,"systemMemoryUsed":814808,"systemMemoryVirtualSize":7494184960,"logInfo":"/var/log/cloudstack/management/management-server.log
> >>> > > > using: 74M\non disk /dev/vda2 mounted on / (14%
> >>> > > >
> >>> > >
> >>> >
> >>>
> full)","systemTotalCpuCycles":15870.180000000002,"systemLoadAverages":[0.02,0.03,0.0],"systemCyclesUsage":[4116569,2615101,2716460424],"dbLocal":false,"usageLocal":false,"systemBootTime":"Jul
> >>> > > > 8, 2022, 2:35:55 AM","kernelVersion":"5.4.0-121-generic"}
> >>> > > > 2022-08-29 17:02:01,723 DEBUG [o.a.c.h.HAManagerImpl]
> >>> > > > (BackgroundTaskPollManager-4:ctx-7abb1b5d) (logid:2320950d) HA
> >>> health
> >>> > > check
> >>> > > > task is running...
> >>> > > > 2022-08-29 17:02:05,417 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>> > > > (Vm-Operations-Cleanup-1:ctx-e741e208) (logid:3f5129e0) VM
> >>> Operation
> >>> > > Thread
> >>> > > > Running
> >>> > > > 2022-08-29 17:02:05,725 DEBUG [o.a.c.h.HAManagerImpl]
> >>> > > > (BackgroundTaskPollManager-4:ctx-8e7f6766) (logid:710cb3b2) HA
> >>> health
> >>> > > check
> >>> > > > task is running...
> >>> > > > 2022-08-29 17:02:08,626 DEBUG [c.c.a.m.AgentManagerImpl]
> >>> > > > (AgentManager-Handler-11:null) (logid:) Ping from Routing host
> >>> > > > 6(srv-3-comp.aaaaaaaa.bbb)
> >>> > > >
> >>> > > > 2022-08-29 17:02:09,727 DEBUG [o.a.c.h.HAManagerImpl]
> >>> > > > (BackgroundTaskPollManager-4:ctx-26a5ff07) (logid:09b3c89a) HA
> >>> health
> >>> > > check
> >>> > > > task is running...
> >>> > > > 2022-08-29 17:02:13,729 DEBUG [o.a.c.h.HAManagerImpl]
> >>> > > > (BackgroundTaskPollManager-4:ctx-b4d4e374) (logid:9048c7a7) HA
> >>> health
> >>> > > check
> >>> > > > task is running...
> >>> > > > 2022-08-29 17:02:17,731 DEBUG [o.a.c.h.HAManagerImpl]
> >>> > > > (BackgroundTaskPollManager-4:ctx-3db3d208) (logid:7bb73f7b) HA
> >>> health
> >>> > > check
> >>> > > > task is running...
> >>> > > > 2022-08-29 17:02:21,734 DEBUG [o.a.c.h.HAManagerImpl]
> >>> > > > (BackgroundTaskPollManager-1:ctx-fca1d248) (logid:1ed5381a) HA
> >>> health
> >>> > > check
> >>> > > > task is running...
> >>> > > > 2022-08-29 17:02:25,737 DEBUG [o.a.c.h.HAManagerImpl]
> >>> > > > (BackgroundTaskPollManager-1:ctx-984e2382) (logid:e6c3d724) HA
> >>> health
> >>> > > check
> >>> > > > task is running...
> >>> > > > 2022-08-29 17:02:25,759 DEBUG [c.c.s.StatsCollector]
> >>> > > > (secstorage-1:ctx-64c610ed) (logid:6603cd38) Verifying image
> >>> storage
> >>> > [1].
> >>> > > > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
> >>> > > > 2022-08-29 17:02:25,763 DEBUG
> [o.a.c.s.SecondaryStorageManagerImpl]
> >>> > > > (secstorage-1:ctx-64c610ed) (logid:6603cd38) Zone [1] is ready to
> >>> > launch
> >>> > > > secondary storage VM.
> >>> > > > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> >>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Zone 1 is ready to
> >>> > launch
> >>> > > > console proxy
> >>> > > > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> >>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Expand console
> proxy
> >>> > > standby
> >>> > > > capacity for zone xxxxxxxxxx
> >>> > > > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> >>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Allocating console
> >>> proxy
> >>> > > > standby capacity for zone [1].
> >>> > > > 2022-08-29 17:02:26,030 INFO [c.c.c.ConsoleProxyManagerImpl]
> >>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Found a stopped
> >>> console
> >>> > > > proxy, starting it. Vm id : 85
> >>> > > > 2022-08-29 17:02:26,031 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) start parameter
> >>> value of
> >>> > > > enterHardwareSetup == <very null> during processing of queued job
> >>> > > > 2022-08-29 17:02:26,037 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Sync job-4289
> >>> execution
> >>> > on
> >>> > > > object VmWorkJobQueue.85
> >>> > > > 2022-08-29 17:02:26,167 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (AsyncJobMgr-Heartbeat-1:ctx-bba6b89d) (logid:f9aed0eb) Execute
> >>> > > sync-queue
> >>> > > > item: SyncQueueItemVO {id:2106, queueId: 2105, contentType:
> >>> AsyncJob,
> >>> > > > contentId: 4289, lastProcessMsid: 90520740759984,
> >>> lastprocessNumber: 2,
> >>> > > > lastProcessTime: Mon Aug 29 17:02:26 CEST 2022, created: Mon Aug
> 29
> >>> > > > 17:02:26 CEST 2022}
> >>> > > > 2022-08-29 17:02:26,167 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (AsyncJobMgr-Heartbeat-1:ctx-bba6b89d) (logid:f9aed0eb) Schedule
> >>> queued
> >>> > > > job-4289
> >>> > > > 2022-08-29 17:02:26,172 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
> >>> (logid:f919f9b3)
> >>> > > Add
> >>> > > > job-4289 into job monitoring
> >>> > > > 2022-08-29 17:02:26,181 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
> >>> (logid:ba10fec5)
> >>> > > > Executing AsyncJobVO: {id:4289, userId: 1, accountId: 1,
> >>> instanceType:
> >>> > > > null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
> >>> > > >
> >>> > >
> >>> >
> >>>
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAVXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
> >>> > > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
> >>> 0,
> >>> > > > result: null, initMsid: 90520740759984, completeMsid: null,
> >>> > lastUpdated:
> >>> > > > null, lastPolled: null, created: Mon Aug 29 17:02:26 CEST 2022,
> >>> > removed:
> >>> > > > null}
> >>> > > > 2022-08-29 17:02:26,181 DEBUG [c.c.v.VmWorkJobDispatcher]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
> >>> (logid:ba10fec5)
> >>> > > Run
> >>> > > > VM work job: com.cloud.vm.VmWorkStart for VM 85, job origin: 3445
> >>> > > > 2022-08-29 17:02:26,183 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb)
> >>> > > > (logid:ba10fec5) Execute VM work job:
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> >>> > > > 2022-08-29 17:02:26,184 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb)
> >>> > > > (logid:ba10fec5) orchestrating VM start for 'v-85-VM'
> >>> > > > com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
> >>> > > > 2022-08-29 17:02:26,192 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb)
> >>> > > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> >>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
> state
> >>> > > > transited from [Stopped] to [Starting] with event
> [StartRequested].
> >>> > VM's
> >>> > > > original host: null, new host: null, host before state
> transition:
> >>> null
> >>> > > > 2022-08-29 17:02:26,192 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb)
> >>> > > > (logid:ba10fec5) Successfully transitioned to start state for VM
> >>> > instance
> >>> > > > {id: "85", name: "v-85-VM", uuid:
> >>> > "110d8620-c029-4ca9-9084-210099725a80",
> >>> > > > type="ConsoleProxy"} reservation id =
> >>> > > 24bf955f-96d8-4f4f-ae33-2cae2f3e0425
> >>> > > > 2022-08-29 17:02:26,198 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb)
> >>> > > > (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts:
> >>> null
> >>> > > > 2022-08-29 17:02:26,200 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb)
> >>> > > > (logid:ba10fec5) VM start attempt #1
> >>> > > > 2022-08-29 17:02:26,203 DEBUG
> [c.c.d.DeploymentPlanningManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb)
> >>> > > > (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
> >>> > > > 2022-08-29 17:02:26,203 DEBUG
> [c.c.d.DeploymentPlanningManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb)
> >>> > > > (logid:ba10fec5) Trying to allocate a host and storage pools from
> >>> dc:1,
> >>> > > > pod:null,cluster:null, requested cpu: 500, requested ram: (1.00
> GB)
> >>> > > > 1073741824
> >>> > > > 2022-08-29 17:02:26,203 DEBUG
> [c.c.d.DeploymentPlanningManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb)
> >>> > > > (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?:
> No
> >>> > > > 2022-08-29 17:02:26,211 DEBUG
> [c.c.d.DeploymentPlanningManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb)
> >>> > > > (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts:
> null
> >>> > > > 2022-08-29 17:02:26,212 DEBUG [c.c.d.FirstFitPlanner]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb)
> >>> > > > (logid:ba10fec5) Searching all possible resources under this
> Zone:
> >>> 1
> >>> > > > 2022-08-29 17:02:26,213 DEBUG [c.c.d.FirstFitPlanner]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb)
> >>> > > > (logid:ba10fec5) Listing clusters in order of aggregate capacity,
> >>> that
> >>> > > have
> >>> > > > (at least one host with) enough CPU and RAM capacity under this
> >>> Zone: 1
> >>> > > > 2022-08-29 17:02:26,216 DEBUG [c.c.d.FirstFitPlanner]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb)
> >>> > > > (logid:ba10fec5) Removing from the clusterId list these clusters
> >>> from
> >>> > > avoid
> >>> > > > set: []
> >>> > > > 2022-08-29 17:02:26,221 DEBUG
> [c.c.d.DeploymentPlanningManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb)
> >>> > > > (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
> >>> > > > 2022-08-29 17:02:26,222 INFO [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested
> >>> with
> >>> > > > Custom[UEFI] Boot Type false
> >>> > > > 2022-08-29 17:02:26,223 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in
> >>> dc: 1
> >>> > > > pod:1 cluster:1
> >>> > > > 2022-08-29 17:02:26,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has
> 4
> >>> > hosts
> >>> > > to
> >>> > > > check for allocation: [Host {"id": "7", "name":
> >>> > > "srv-5-comp.aaaaaaaa.bbb",
> >>> > > > "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6",
> "type"="Routing"},
> >>> Host
> >>> > > > {"id": "1", "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> >>> > > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}, Host
> >>> {"id":
> >>> > > "6",
> >>> > > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> >>> > > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
> >>> {"id":
> >>> > > "5",
> >>> > > > "name": "srv-4-comp.aaaaaaaa.bbb", "uuid":
> >>> > > > "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}]
> >>> > > > 2022-08-29 17:02:26,231 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for
> >>> allocation
> >>> > > > after prioritization: [Host {"id": "7", "name":
> >>> > > "srv-5-comp.aaaaaaaa.bbb",
> >>> > > > "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6",
> "type"="Routing"},
> >>> Host
> >>> > > > {"id": "1", "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> >>> > > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}, Host
> >>> {"id":
> >>> > > "6",
> >>> > > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> >>> > > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
> >>> {"id":
> >>> > > "5",
> >>> > > > "name": "srv-4-comp.aaaaaaaa.bbb", "uuid":
> >>> > > > "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}]
> >>> > > > 2022-08-29 17:02:26,231 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for
> >>> speed=500Mhz,
> >>> > > > Ram=1024 MB
> >>> > > > 2022-08-29 17:02:26,231 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
> >>> > > > srv-5-comp.aaaaaaaa.bbb, uuid:
> >>> 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is
> >>> > > KVM
> >>> > > > hypervisor type, no max guest limit check needed
> >>> > > > 2022-08-29 17:02:26,234 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu
> >>> capability
> >>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
> >>> speed:
> >>> > 500
> >>> > > > 2022-08-29 17:02:26,234 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7
> has
> >>> > enough
> >>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> >>> > 1073741824 ,
> >>> > > > cpuOverprovisioningFactor: 1.0
> >>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
> >>> CPU:
> >>> > > 127200
> >>> > > > and CPU after applying overprovisioning: 127200
> >>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 ,
> >>> Requested
> >>> > > CPU:
> >>> > > > 500
> >>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
> >>> > > > 189373743104 , Requested RAM: (1.00 GB) 1073741824
> >>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
> and
> >>> RAM
> >>> > > > available
> >>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
> >>> from
> >>> > > host:
> >>> > > > 7, used: 90100, reserved: 0, actual total: 127200, total with
> >>> > > > overprovisioning: 127200; requested
> >>> cpu:500,alloc_from_last_host?:false
> >>> > > > ,considerReservedCapacity?: true
> >>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
> >>> from
> >>> > > host:
> >>> > > > 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total:
> >>> (250.27
> >>> > > GB)
> >>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
> >>> > alloc_from_last_host?:
> >>> > > > false , considerReservedCapacity?: true
> >>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> >>> > adding
> >>> > > to
> >>> > > > list: 7
> >>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
> >>> > > > srv-2-comp.aaaaaaaa.bbb, uuid:
> >>> 883f7f62-55f2-42f4-8242-16b6847c4204} is
> >>> > > KVM
> >>> > > > hypervisor type, no max guest limit check needed
> >>> > > > 2022-08-29 17:02:26,239 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu
> >>> capability
> >>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
> >>> speed:
> >>> > 500
> >>> > > > 2022-08-29 17:02:26,239 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1
> has
> >>> > enough
> >>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> >>> > 1073741824 ,
> >>> > > > cpuOverprovisioningFactor: 1.0
> >>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
> >>> CPU:
> >>> > > 127200
> >>> > > > and CPU after applying overprovisioning: 127200
> >>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 ,
> >>> Requested
> >>> > > CPU:
> >>> > > > 500
> >>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
> >>> > > > 229802639360 , Requested RAM: (1.00 GB) 1073741824
> >>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
> and
> >>> RAM
> >>> > > > available
> >>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
> >>> from
> >>> > > host:
> >>> > > > 1, used: 48200, reserved: 0, actual total: 127200, total with
> >>> > > > overprovisioning: 127200; requested
> >>> cpu:500,alloc_from_last_host?:false
> >>> > > > ,considerReservedCapacity?: true
> >>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
> >>> from
> >>> > > host:
> >>> > > > 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total:
> >>> (250.27
> >>> > > GB)
> >>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
> >>> > alloc_from_last_host?:
> >>> > > > false , considerReservedCapacity?: true
> >>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> >>> > adding
> >>> > > to
> >>> > > > list: 1
> >>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
> >>> > > > srv-3-comp.aaaaaaaa.bbb, uuid:
> >>> 6858713b-7e17-46d8-854d-24a13f9c7001} is
> >>> > > KVM
> >>> > > > hypervisor type, no max guest limit check needed
> >>> > > > 2022-08-29 17:02:26,243 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu
> >>> capability
> >>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
> >>> speed:
> >>> > 500
> >>> > > > 2022-08-29 17:02:26,243 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6
> has
> >>> > enough
> >>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> >>> > 1073741824 ,
> >>> > > > cpuOverprovisioningFactor: 1.0
> >>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
> >>> CPU:
> >>> > > 127200
> >>> > > > and CPU after applying overprovisioning: 127200
> >>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 ,
> >>> Requested
> >>> > > CPU:
> >>> > > > 500
> >>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
> >>> > > > 204032835584 , Requested RAM: (1.00 GB) 1073741824
> >>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
> and
> >>> RAM
> >>> > > > available
> >>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
> >>> from
> >>> > > host:
> >>> > > > 6, used: 42900, reserved: 0, actual total: 127200, total with
> >>> > > > overprovisioning: 127200; requested
> >>> cpu:500,alloc_from_last_host?:false
> >>> > > > ,considerReservedCapacity?: true
> >>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
> >>> from
> >>> > > host:
> >>> > > > 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total:
> >>> (250.27
> >>> > > GB)
> >>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
> >>> > alloc_from_last_host?:
> >>> > > > false , considerReservedCapacity?: true
> >>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> >>> > adding
> >>> > > to
> >>> > > > list: 6
> >>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
> >>> > > > srv-4-comp.aaaaaaaa.bbb, uuid:
> >>> 134c625d-15ab-461a-a38d-36fb0b979b5d} is
> >>> > > KVM
> >>> > > > hypervisor type, no max guest limit check needed
> >>> > > > 2022-08-29 17:02:26,247 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu
> >>> capability
> >>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
> >>> speed:
> >>> > 500
> >>> > > > 2022-08-29 17:02:26,247 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5
> has
> >>> > enough
> >>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> >>> > 1073741824 ,
> >>> > > > cpuOverprovisioningFactor: 1.0
> >>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
> >>> CPU:
> >>> > > 127200
> >>> > > > and CPU after applying overprovisioning: 127200
> >>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 ,
> >>> Requested
> >>> > > > CPU: 500
> >>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
> >>> > > > 251009040384 , Requested RAM: (1.00 GB) 1073741824
> >>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
> and
> >>> RAM
> >>> > > > available
> >>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
> >>> from
> >>> > > host:
> >>> > > > 5, used: 11100, reserved: 0, actual total: 127200, total with
> >>> > > > overprovisioning: 127200; requested
> >>> cpu:500,alloc_from_last_host?:false
> >>> > > > ,considerReservedCapacity?: true
> >>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
> >>> from
> >>> > > host:
> >>> > > > 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total:
> >>> (250.27
> >>> > > GB)
> >>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
> >>> > alloc_from_last_host?:
> >>> > > > false , considerReservedCapacity?: true
> >>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> >>> > adding
> >>> > > to
> >>> > > > list: 5
> >>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb
> >>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator
> >>> returning 4
> >>> > > > suitable hosts
> >>> > > > 2022-08-29 17:02:26,260 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb)
> >>> > > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> >>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
> state
> >>> > > > transited from [Starting] to [Stopped] with event
> >>> [OperationFailed].
> >>> > VM's
> >>> > > > original host: null, new host: null, host before state
> transition:
> >>> null
> >>> > > > 2022-08-29 17:02:26,260 WARN [c.c.v.VirtualMachineManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb)
> >>> > > > (logid:ba10fec5) Unable to orchestrate start VM instance {id:
> "85",
> >>> > name:
> >>> > > > "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
> >>> > > > type="ConsoleProxy"} due to [Unable to create deployment, no
> usable
> >>> > > volumes
> >>> > > > found for the VM: 85].
> >>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to create
> >>> > > > deployment, no usable volumes found for the VM: 85
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
> >>> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> >>> > Source)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> >>> > > > at
> >>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> >>> > > > at
> >>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
> >>> > > > 2022-08-29 17:02:26,261 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb)
> >>> > > > (logid:ba10fec5) Done executing VM work job:
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> >>> > > > 2022-08-29 17:02:26,262 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb)
> >>> > > > (logid:ba10fec5) Complete async job-4289, jobStatus: FAILED,
> >>> > resultCode:
> >>> > > 0,
> >>> > > > result:
> >>> > > >
> >>> > >
> >>> >
> >>>
> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODUiLCBuYW1lOiAidi04NS1WTSIsIHV1aWQ6ICIxMTBkODYyMC1jMDI5LTRjYTktOTA4NC0yMTAwOTk3MjVhODAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODVdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
> >>> > > > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb)
> >>> > > > (logid:ba10fec5) Publish async job-4289 complete on message bus
> >>> > > > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb)
> >>> > > > (logid:ba10fec5) Wake up jobs related to job-4289
> >>> > > > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb)
> >>> > > > (logid:ba10fec5) Update db status for job-4289
> >>> > > > 2022-08-29 17:02:26,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289
> ctx-214097cb)
> >>> > > > (logid:ba10fec5) Wake up jobs joined with job-4289 and disjoin
> all
> >>> > > subjobs
> >>> > > > created from job- 4289
> >>> > > > 2022-08-29 17:02:26,269 DEBUG [c.c.v.VmWorkJobDispatcher]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
> >>> (logid:ba10fec5)
> >>> > > > Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 85,
> >>> job
> >>> > > > origin: 3445
> >>> > > > 2022-08-29 17:02:26,270 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
> >>> (logid:ba10fec5)
> >>> > > > Done executing com.cloud.vm.VmWorkStart for job-4289
> >>> > > > 2022-08-29 17:02:26,271 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
> >>> (logid:ba10fec5)
> >>> > > > Remove job-4289 from job monitoring
> >>> > > > 2022-08-29 17:02:26,277 WARN [c.c.c.ConsoleProxyManagerImpl]
> >>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Unable to allocate
> >>> > console
> >>> > > > proxy standby capacity for zone [1] due to [Unable to orchestrate
> >>> start
> >>> > > VM
> >>> > > > instance {id: "85", name: "v-85-VM", uuid:
> >>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due
> to
> >>> > > [Unable
> >>> > > > to create deployment, no usable volumes found for the VM: 85].].
> >>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to
> >>> orchestrate
> >>> > > > start VM instance {id: "85", name: "v-85-VM", uuid:
> >>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due
> to
> >>> > > [Unable
> >>> > > > to create deployment, no usable volumes found for the VM: 85].
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> >>> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> >>> > Source)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> >>> > > > at
> >>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> >>> > > > at
> >>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >>> > > > at
> >>> > > >
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
> >>> > > > 2022-08-29 17:02:26,277 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> >>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) received console
> >>> proxy
> >>> > > alert
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > Am Mo., 29. Aug. 2022 um 16:30 Uhr schrieb Slavka Peleva <
> >>> > > > slavkap@storpool.com>:
> >>> > > >
> >>> > > >> Thanks, Chris, for the information!
> >>> > > >>
> >>> > > >> Are you using custom offerings for the system VMs? Can you share
> >>> more
> >>> > > >> info if yes?
> >>> > > >>
> >>> > > >> About the log, it's not the complete log from where the
> deployment
> >>> > > >> started. The deployment begins a bit earlier than what you
> >>> shared. Can
> >>> > > you
> >>> > > >> search the management log for `Sync job-3539 execution on object
> >>> > > >> VmWorkJobQueue` to get the `logid` for the volume creation
> >>> (coloured
> >>> > in
> >>> > > >> blue)? And if possible, share the logs with the found log ID
> >>> > > >>
> >>> > > >> Here is an example from my logs:
> >>> > > >>
> >>> > > >> 2022-08-29 17:07:49,114 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > >> (consoleproxy-1:ctx-a952a3dc) (logid:89157bb9) Sync job-60484
> >>> > execution
> >>> > > >> on object VmWorkJobQueue.4904
> >>> > > >>
> >>> > > >> 2022-08-29 17:07:50,816 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> >>> > > >> (Work-Job-Executor-5:ctx-56172581 job-60479/job-60484
> >>> ctx-a900986d)
> >>> > > >> (logid:95d750ae) Execute VM work job:
> >>> > > >>
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":4904,"handlerName":"VirtualMachineManagerImpl"}
> >>> > > >> 2022-08-29 17:07:50,817 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>> > > >> (Work-Job-Executor-5:ctx-56172581 job-60479/job-60484
> >>> ctx-a900986d)
> >>> > > >> (logid:95d750ae) orchestrating VM start for 'v-4904-VM'
> >>> > > >> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
> >>> > > >>
> >>> > > >>
> >>> > > >> On Mon, Aug 29, 2022 at 2:29 PM Vash_X@gmx.de <Va...@gmx.de>
> >>> wrote:
> >>> > > >>
> >>> > > >>> Hi Slavka,
> >>> > > >>>
> >>> > > >>> thats what i did after your suggestion / informatin and where
> the
> >>> > error
> >>> > > >>> occured.
> >>> > > >>> Deleted / Destroyed the consoleproxy and waited till
> recreation.
> >>> > Which
> >>> > > >>> didn't worked throwing the error i pasted above.
> >>> > > >>> Sorry for beeing so unspecific.
> >>> > > >>>
> >>> > > >>> Regards,
> >>> > > >>> Chris
> >>> > > >>>
> >>> > > >>> Am Mo., 29. Aug. 2022 um 12:46 Uhr schrieb Slavka Peleva <
> >>> > > >>> slavkap@storpool.com>:
> >>> > > >>>
> >>> > > >>>> Hi Chris,
> >>> > > >>>>
> >>> > > >>>> I mean to recreate the system VMs by destroying them.
> >>> > > >>>>
> >>> > > >>>> On Mon, Aug 29, 2022 at 12:38 PM Vash_X@gmx.de <Vash_X@gmx.de
> >
> >>> > wrote:
> >>> > > >>>>
> >>> > > >>>>> Hi Slavka,
> >>> > > >>>>>
> >>> > > >>>>> didn't tried to restart the SystemVMs. Reading through the
> >>> docs i
> >>> > had
> >>> > > >>>>> the impression that "live-patching" would be enough for
> >>> this....
> >>> > > >>>>> Anyway - I tried my luck with the console proxy.
> >>> > > >>>>>
> >>> > > >>>>> Which failed utterly. Not starting as it isn't finding any
> >>> useable
> >>> > > >>>>> storage pool. - logs at the end of the mail.
> >>> > > >>>>>
> >>> > > >>>>> I tried deploying a VM using the same storage pool which
> worked
> >>> > > >>>>> wonderfully.
> >>> > > >>>>> There i noticed that all my custome storage-offerings are
> >>> unuseable
> >>> > > at
> >>> > > >>>>> the moment. They are not accepting any custome storage sizes
> >>> > > anymore...
> >>> > > >>>>> Throwing failures "VM Creation failed. Volume size: -1GB is
> >>> out of
> >>> > > allowed
> >>> > > >>>>> tange. Max:3072 Min:1"...
> >>> > > >>>>> So this is also currently broken....
> >>> > > >>>>>
> >>> > > >>>>> 2022-08-29 10:51:56,189 DEBUG [c.c.v.VmWorkJobDispatcher]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
> >>> > > (logid:ba10fec5) Run
> >>> > > >>>>> VM work job: com.cloud.vm.VmWorkStart for VM 83, job origin:
> >>> 3445
> >>> > > >>>>> 2022-08-29 10:51:56,191 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751)
> >>> > > >>>>> (logid:ba10fec5) Execute VM work job:
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
> >>> > > >>>>> 2022-08-29 10:51:56,192 DEBUG
> [c.c.v.VirtualMachineManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751)
> >>> > > >>>>> (logid:ba10fec5) orchestrating VM start for 'v-83-VM'
> >>> > > >>>>> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to
> null
> >>> > > >>>>> 2022-08-29 10:51:56,198 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751)
> >>> > > >>>>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM",
> uuid:
> >>> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
> >>> state
> >>> > > >>>>> transited from [Stopped] to [Starting] with event
> >>> [StartRequested].
> >>> > > VM's
> >>> > > >>>>> original host: null, new host: null, host before state
> >>> transition:
> >>> > > null
> >>> > > >>>>> 2022-08-29 10:51:56,198 DEBUG
> [c.c.v.VirtualMachineManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751)
> >>> > > >>>>> (logid:ba10fec5) Successfully transitioned to start state for
> >>> VM
> >>> > > instance
> >>> > > >>>>> {id: "83", name: "v-83-VM", uuid:
> >>> > > "b1d1f9c6-eafe-4527-9335-e649c646aab0",
> >>> > > >>>>> type="ConsoleProxy"} reservation id =
> >>> > > 6641ccc0-bcef-4b1a-a1f7-8dd8000626ad
> >>> > > >>>>> 2022-08-29 10:51:56,203 DEBUG
> [c.c.v.VirtualMachineManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751)
> >>> > > >>>>> (logid:ba10fec5) Deploy avoids pods: null, clusters: null,
> >>> hosts:
> >>> > > null
> >>> > > >>>>> 2022-08-29 10:51:56,206 DEBUG
> [c.c.v.VirtualMachineManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751)
> >>> > > >>>>> (logid:ba10fec5) VM start attempt #1
> >>> > > >>>>> 2022-08-29 10:51:56,207 DEBUG
> >>> [c.c.d.DeploymentPlanningManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751)
> >>> > > >>>>> (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
> >>> > > >>>>> 2022-08-29 10:51:56,207 DEBUG
> >>> [c.c.d.DeploymentPlanningManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751)
> >>> > > >>>>> (logid:ba10fec5) Trying to allocate a host and storage pools
> >>> from
> >>> > > dc:1,
> >>> > > >>>>> pod:null,cluster:null, requested cpu: 500, requested ram:
> >>> (1.00 GB)
> >>> > > >>>>> 1073741824
> >>> > > >>>>> 2022-08-29 10:51:56,207 DEBUG
> >>> [c.c.d.DeploymentPlanningManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751)
> >>> > > >>>>> (logid:ba10fec5) Is ROOT volume READY (pool already
> >>> allocated)?: No
> >>> > > >>>>> 2022-08-29 10:51:56,214 DEBUG
> >>> [c.c.d.DeploymentPlanningManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751)
> >>> > > >>>>> (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts:
> >>> null
> >>> > > >>>>> 2022-08-29 10:51:56,215 DEBUG [c.c.d.FirstFitPlanner]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751)
> >>> > > >>>>> (logid:ba10fec5) Searching all possible resources under this
> >>> Zone:
> >>> > 1
> >>> > > >>>>> 2022-08-29 10:51:56,216 DEBUG [c.c.d.FirstFitPlanner]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751)
> >>> > > >>>>> (logid:ba10fec5) Listing clusters in order of aggregate
> >>> capacity,
> >>> > > that have
> >>> > > >>>>> (at least one host with) enough CPU and RAM capacity under
> this
> >>> > > Zone: 1
> >>> > > >>>>> 2022-08-29 10:51:56,219 DEBUG [c.c.d.FirstFitPlanner]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751)
> >>> > > >>>>> (logid:ba10fec5) Removing from the clusterId list these
> >>> clusters
> >>> > > from avoid
> >>> > > >>>>> set: []
> >>> > > >>>>> 2022-08-29 10:51:56,224 DEBUG
> >>> [c.c.d.DeploymentPlanningManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751)
> >>> > > >>>>> (logid:ba10fec5) Checking resources in Cluster: 1 under Pod:
> 1
> >>> > > >>>>> 2022-08-29 10:51:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is
> >>> requested
> >>> > > with
> >>> > > >>>>> Custom[UEFI] Boot Type false
> >>> > > >>>>> 2022-08-29 10:51:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts
> in
> >>> > dc: 1
> >>> > > >>>>> pod:1 cluster:1
> >>> > > >>>>> 2022-08-29 10:51:56,227 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator
> >>> has 4
> >>> > > hosts to
> >>> > > >>>>> check for allocation: [Host {"id": "7", "name":
> >>> > > "srv-5-comp.blabla.blub",
> >>> > > >>>>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6",
> >>> "type"="Routing"},
> >>> > > Host
> >>> > > >>>>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
> >>> > > >>>>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"},
> Host
> >>> > > {"id": "6",
> >>> > > >>>>> "name": "srv-3-comp.blabla.blub", "uuid":
> >>> > > >>>>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"},
> Host
> >>> > > {"id": "1",
> >>> > > >>>>> "name": "srv-2-comp.blabla.blub", "uuid":
> >>> > > >>>>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> >>> > > >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for
> >>> > > allocation
> >>> > > >>>>> after prioritization: [Host {"id": "7", "name":
> >>> > > "srv-5-comp.blabla.blub",
> >>> > > >>>>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6",
> >>> "type"="Routing"},
> >>> > > Host
> >>> > > >>>>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
> >>> > > >>>>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"},
> Host
> >>> > > {"id": "6",
> >>> > > >>>>> "name": "srv-3-comp.blabla.blub", "uuid":
> >>> > > >>>>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"},
> Host
> >>> > > {"id": "1",
> >>> > > >>>>> "name": "srv-2-comp.blabla.blub", "uuid":
> >>> > > >>>>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> >>> > > >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for
> >>> > speed=500Mhz,
> >>> > > >>>>> Ram=1024 MB
> >>> > > >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
> >>> > > >>>>> srv-5-comp.blabla.blub, uuid:
> >>> 8c753ce1-0253-49c9-9c7f-1f971a719ce6}
> >>> > > is KVM
> >>> > > >>>>> hypervisor type, no max guest limit check needed
> >>> > > >>>>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu
> >>> > capability
> >>> > > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and
> requested
> >>> > > speed: 500
> >>> > > >>>>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host:
> 7
> >>> has
> >>> > > enough
> >>> > > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> >>> > > 1073741824 ,
> >>> > > >>>>> cpuOverprovisioningFactor: 1.0
> >>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual
> total
> >>> > CPU:
> >>> > > 127200
> >>> > > >>>>> and CPU after applying overprovisioning: 127200
> >>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 ,
> >>> > > Requested CPU:
> >>> > > >>>>> 500
> >>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37
> >>> GB)
> >>> > > >>>>> 189373743104 , Requested RAM: (1.00 GB) 1073741824
> >>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough
> CPU
> >>> and
> >>> > > RAM
> >>> > > >>>>> available
> >>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc
> CPU
> >>> > from
> >>> > > host:
> >>> > > >>>>> 7, used: 90100, reserved: 0, actual total: 127200, total with
> >>> > > >>>>> overprovisioning: 127200; requested
> >>> > > cpu:500,alloc_from_last_host?:false
> >>> > > >>>>> ,considerReservedCapacity?: true
> >>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc
> MEM
> >>> > from
> >>> > > host:
> >>> > > >>>>> 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0,
> total:
> >>> > > (250.27 GB)
> >>> > > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
> >>> > > alloc_from_last_host?:
> >>> > > >>>>> false , considerReservedCapacity?: true
> >>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable
> >>> host,
> >>> > > adding to
> >>> > > >>>>> list: 7
> >>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
> >>> > > >>>>> srv-4-comp.blabla.blub, uuid:
> >>> 134c625d-15ab-461a-a38d-36fb0b979b5d}
> >>> > > is KVM
> >>> > > >>>>> hypervisor type, no max guest limit check needed
> >>> > > >>>>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu
> >>> > capability
> >>> > > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and
> requested
> >>> > > speed: 500
> >>> > > >>>>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host:
> 5
> >>> has
> >>> > > enough
> >>> > > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> >>> > > 1073741824 ,
> >>> > > >>>>> cpuOverprovisioningFactor: 1.0
> >>> > > >>>>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual
> total
> >>> > CPU:
> >>> > > 127200
> >>> > > >>>>> and CPU after applying overprovisioning: 127200
> >>> > > >>>>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 ,
> >>> > > Requested
> >>> > > >>>>> CPU: 500
> >>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77
> >>> GB)
> >>> > > >>>>> 251009040384 , Requested RAM: (1.00 GB) 1073741824
> >>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough
> CPU
> >>> and
> >>> > > RAM
> >>> > > >>>>> available
> >>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc
> CPU
> >>> > from
> >>> > > host:
> >>> > > >>>>> 5, used: 11100, reserved: 0, actual total: 127200, total with
> >>> > > >>>>> overprovisioning: 127200; requested
> >>> > > cpu:500,alloc_from_last_host?:false
> >>> > > >>>>> ,considerReservedCapacity?: true
> >>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc
> MEM
> >>> > from
> >>> > > host:
> >>> > > >>>>> 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0,
> total:
> >>> > > (250.27 GB)
> >>> > > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
> >>> > > alloc_from_last_host?:
> >>> > > >>>>> false , considerReservedCapacity?: true
> >>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable
> >>> host,
> >>> > > adding to
> >>> > > >>>>> list: 5
> >>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
> >>> > > >>>>> srv-3-comp.blabla.blub, uuid:
> >>> 6858713b-7e17-46d8-854d-24a13f9c7001}
> >>> > > is KVM
> >>> > > >>>>> hypervisor type, no max guest limit check needed
> >>> > > >>>>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu
> >>> > capability
> >>> > > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and
> requested
> >>> > > speed: 500
> >>> > > >>>>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host:
> 6
> >>> has
> >>> > > enough
> >>> > > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> >>> > > 1073741824 ,
> >>> > > >>>>> cpuOverprovisioningFactor: 1.0
> >>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual
> total
> >>> > CPU:
> >>> > > 127200
> >>> > > >>>>> and CPU after applying overprovisioning: 127200
> >>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 ,
> >>> > > Requested CPU:
> >>> > > >>>>> 500
> >>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02
> >>> GB)
> >>> > > >>>>> 204032835584 , Requested RAM: (1.00 GB) 1073741824
> >>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough
> CPU
> >>> and
> >>> > > RAM
> >>> > > >>>>> available
> >>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc
> CPU
> >>> > from
> >>> > > host:
> >>> > > >>>>> 6, used: 42900, reserved: 0, actual total: 127200, total with
> >>> > > >>>>> overprovisioning: 127200; requested
> >>> > > cpu:500,alloc_from_last_host?:false
> >>> > > >>>>> ,considerReservedCapacity?: true
> >>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc
> MEM
> >>> > from
> >>> > > host:
> >>> > > >>>>> 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0,
> total:
> >>> > > (250.27 GB)
> >>> > > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
> >>> > > alloc_from_last_host?:
> >>> > > >>>>> false , considerReservedCapacity?: true
> >>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable
> >>> host,
> >>> > > adding to
> >>> > > >>>>> list: 6
> >>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
> >>> > > >>>>> srv-2-comp.blabla.blub, uuid:
> >>> 883f7f62-55f2-42f4-8242-16b6847c4204}
> >>> > > is KVM
> >>> > > >>>>> hypervisor type, no max guest limit check needed
> >>> > > >>>>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu
> >>> > capability
> >>> > > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and
> requested
> >>> > > speed: 500
> >>> > > >>>>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host:
> 1
> >>> has
> >>> > > enough
> >>> > > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> >>> > > 1073741824 ,
> >>> > > >>>>> cpuOverprovisioningFactor: 1.0
> >>> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual
> total
> >>> > CPU:
> >>> > > 127200
> >>> > > >>>>> and CPU after applying overprovisioning: 127200
> >>> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 ,
> >>> > > Requested CPU:
> >>> > > >>>>> 500
> >>> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02
> >>> GB)
> >>> > > >>>>> 229802639360 , Requested RAM: (1.00 GB) 1073741824
> >>> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough
> CPU
> >>> and
> >>> > > RAM
> >>> > > >>>>> available
> >>> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc
> CPU
> >>> > from
> >>> > > host:
> >>> > > >>>>> 1, used: 48200, reserved: 0, actual total: 127200, total with
> >>> > > >>>>> overprovisioning: 127200; requested
> >>> > > cpu:500,alloc_from_last_host?:false
> >>> > > >>>>> ,considerReservedCapacity?: true
> >>> > > >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc
> MEM
> >>> > from
> >>> > > host:
> >>> > > >>>>> 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0,
> total:
> >>> > > (250.27 GB)
> >>> > > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
> >>> > > alloc_from_last_host?:
> >>> > > >>>>> false , considerReservedCapacity?: true
> >>> > > >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable
> >>> host,
> >>> > > adding to
> >>> > > >>>>> list: 1
> >>> > > >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751
> >>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator
> >>> > returning 4
> >>> > > >>>>> suitable hosts
> >>> > > >>>>> 2022-08-29 10:51:56,260 DEBUG [c.c.c.CapacityManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751)
> >>> > > >>>>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM",
> uuid:
> >>> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
> >>> state
> >>> > > >>>>> transited from [Starting] to [Stopped] with event
> >>> > [OperationFailed].
> >>> > > VM's
> >>> > > >>>>> original host: null, new host: null, host before state
> >>> transition:
> >>> > > null
> >>> > > >>>>> 2022-08-29 10:51:56,260 WARN
> [c.c.v.VirtualMachineManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751)
> >>> > > >>>>> (logid:ba10fec5) Unable to orchestrate start VM instance {id:
> >>> "83",
> >>> > > name:
> >>> > > >>>>> "v-83-VM", uuid: "b1d1f9c6-eafe-4527-9335-e649c646aab0",
> >>> > > >>>>> type="ConsoleProxy"} due to [Unable to create deployment, no
> >>> usable
> >>> > > volumes
> >>> > > >>>>> found for the VM: 83].
> >>> > > >>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to
> >>> create
> >>> > > >>>>> deployment, no usable volumes found for the VM: 83
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
> >>> > > >>>>> at
> >>> jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> >>> > > >>>>> Source)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>> > > >>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> >>> > > >>>>> at
> >>> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >>> > > >>>>> at java.base/java.lang.Thread.run(Thread.java:829)
> >>> > > >>>>> 2022-08-29 10:51:56,262 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751)
> >>> > > >>>>> (logid:ba10fec5) Done executing VM work job:
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
> >>> > > >>>>> 2022-08-29 10:51:56,263 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751)
> >>> > > >>>>> (logid:ba10fec5) Complete async job-3539, jobStatus: FAILED,
> >>> > > resultCode: 0,
> >>> > > >>>>> result:
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODMiLCBuYW1lOiAidi04My1WTSIsIHV1aWQ6ICJiMWQxZjljNi1lYWZlLTQ1MjctOTMzNS1lNjQ5YzY0NmFhYjAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODNdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
> >>> > > >>>>> 2022-08-29 10:51:56,264 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751)
> >>> > > >>>>> (logid:ba10fec5) Publish async job-3539 complete on message
> bus
> >>> > > >>>>> 2022-08-29 10:51:56,264 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751)
> >>> > > >>>>> (logid:ba10fec5) Wake up jobs related to job-3539
> >>> > > >>>>> 2022-08-29 10:51:56,264 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751)
> >>> > > >>>>> (logid:ba10fec5) Update db status for job-3539
> >>> > > >>>>> 2022-08-29 10:51:56,266 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> >>> ctx-073f2751)
> >>> > > >>>>> (logid:ba10fec5) Wake up jobs joined with job-3539 and
> disjoin
> >>> all
> >>> > > subjobs
> >>> > > >>>>> created from job- 3539
> >>> > > >>>>> 2022-08-29 10:51:56,276 DEBUG [c.c.v.VmWorkJobDispatcher]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
> >>> > > (logid:ba10fec5) Done
> >>> > > >>>>> with run of VM work job: com.cloud.vm.VmWorkStart for VM 83,
> >>> job
> >>> > > origin:
> >>> > > >>>>> 3445
> >>> > > >>>>> 2022-08-29 10:51:56,276 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
> >>> > > (logid:ba10fec5) Done
> >>> > > >>>>> executing com.cloud.vm.VmWorkStart for job-3539
> >>> > > >>>>> 2022-08-29 10:51:56,278 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
> >>> > > (logid:ba10fec5)
> >>> > > >>>>> Remove job-3539 from job monitoring
> >>> > > >>>>> 2022-08-29 10:51:56,285 WARN [c.c.c.ConsoleProxyManagerImpl]
> >>> > > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Unable to
> >>> allocate
> >>> > > console
> >>> > > >>>>> proxy standby capacity for zone [1] due to [Unable to
> >>> orchestrate
> >>> > > start VM
> >>> > > >>>>> instance {id: "83", name: "v-83-VM", uuid:
> >>> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
> >>> due to
> >>> > > [Unable
> >>> > > >>>>> to create deployment, no usable volumes found for the VM:
> >>> 83].].
> >>> > > >>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to
> >>> > > orchestrate
> >>> > > >>>>> start VM instance {id: "83", name: "v-83-VM", uuid:
> >>> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
> >>> due to
> >>> > > [Unable
> >>> > > >>>>> to create deployment, no usable volumes found for the VM:
> 83].
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> >>> > > >>>>> at
> >>> jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> >>> > > >>>>> Source)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>> > > >>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> >>> > > >>>>> at
> >>> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >>> > > >>>>> at
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >>> > > >>>>> at java.base/java.lang.Thread.run(Thread.java:829)
> >>> > > >>>>> 2022-08-29 10:51:56,287 DEBUG
> [c.c.a.ConsoleProxyAlertAdapter]
> >>> > > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) received
> console
> >>> > > proxy alert
> >>> > > >>>>> 2022-08-29 10:51:56,288 DEBUG
> [c.c.a.ConsoleProxyAlertAdapter]
> >>> > > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Console proxy
> >>> > creation
> >>> > > >>>>> failure, zone: blabla
> >>> > > >>>>> 2022-08-29 10:51:56,289 WARN [c.c.a.AlertManagerImpl]
> >>> > > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) alertType=[10]
> >>> > > >>>>> dataCenterId=[1] podId=[null] clusterId=[null]
> message=[Console
> >>> > proxy
> >>> > > >>>>> creation failure. zone: blabla, error details: Unable to
> >>> > orchestrate
> >>> > > start
> >>> > > >>>>> VM instance {id: "83", name: "v-83-VM", uuid:
> >>> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
> >>> due to
> >>> > > [Unable
> >>> > > >>>>> to create deployment, no usable volumes found for the VM:
> >>> 83].].
> >>> > > >>>>>
> >>> > > >>>>>
> >>> > > >>>>>
> >>> > > >>>>>
> >>> > > >>>>> Am Mo., 29. Aug. 2022 um 09:40 Uhr schrieb Slavka Peleva
> >>> > > >>>>> <sl...@storpool.com.invalid>:
> >>> > > >>>>>
> >>> > > >>>>>> Hi Chris,
> >>> > > >>>>>>
> >>> > > >>>>>> Did you recreate the system VMs? In the 4.17 version, the
> >>> > > >>>>>> `systemvm.iso` is
> >>> > > >>>>>> deprecated. In its place, you should have the file on agents
> >>> > > >>>>>> `/usr/share/cloudstack-common/vms/agent.zip`.
> >>> > > >>>>>> Can you share the complete log if the system VMs are newly
> >>> created
> >>> > > >>>>>> after
> >>> > > >>>>>> the upgrade?
> >>> > > >>>>>>
> >>> > > >>>>>> Best regards,
> >>> > > >>>>>> Slavka
> >>> > > >>>>>>
> >>> > > >>>>>> On Sun, Aug 28, 2022 at 2:16 PM Vash_X@gmx.de <
> Vash_X@gmx.de>
> >>> > > wrote:
> >>> > > >>>>>>
> >>> > > >>>>>> > Hi everyone,
> >>> > > >>>>>> >
> >>> > > >>>>>> > faceing some challanges again after upgradeing to 4.17.
> >>> > > >>>>>> > Did as explained in the docs.
> >>> > > >>>>>> >
> >>> > > >>>>>> > Afterwards I am now not able to migrate system VMs to
> >>> different
> >>> > > >>>>>> hosts.
> >>> > > >>>>>> > Getting the error
> >>> > > >>>>>> > "Exception during migrate: org.libvirt.LibvirtException:
> >>> Cannot
> >>> > > >>>>>> access
> >>> > > >>>>>> > storage file
> >>> '/usr/share/cloudstack-common/vms/systemvm.iso': No
> >>> > > >>>>>> such file
> >>> > > >>>>>> > or directory"
> >>> > > >>>>>> >
> >>> > > >>>>>> > Checked the exitstence of the file on the servers:
> >>> > > >>>>>> > Mangement-Server has a "systemvm.iso.bak"
> >>> > > >>>>>> > Hosts - no "systemvm.iso" at all present.
> >>> > > >>>>>> >
> >>> > > >>>>>> > Maybe someone has an idea on how to move on with this?
> >>> > > >>>>>> >
> >>> > > >>>>>> > Regards,
> >>> > > >>>>>> > Chris
> >>> > > >>>>>> >
> >>> > > >>>>>>
> >>> > > >>>>>
> >>> > >
> >>> >
> >>>
> >>
>
Re: Error migrate System VMs after upgrageing to 4.17
Posted by Nicolas Vazquez <Ni...@shapeblue.com>.
Thanks Chris,
Can you please open a Github issue for this problem?
Regards,
Nicolas Vazquez
________________________________
From: Vash_X@gmx.de <Va...@gmx.de>
Sent: Friday, September 2, 2022 7:15:15 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>; Slavka Peleva <sl...@storpool.com>; Wei ZHOU <us...@gmail.com>
Subject: Re: Error migrate System VMs after upgrageing to 4.17
Hi,
time for a short update.
Today i tried my luck by switching the global setting
*system.vm.public.ip.reservation.mode.strictness* from true to false.
Afterwards i was able to create a systemvm and migrate it to another
server. --> @Slavka Peleva <sl...@storpool.com> so the initial "problem"
of not finding systemvm.iso seems to be solved.
The used public IP "belongs" to a ip range reserved for the domain "Root" (
[Root] in WebGUI Style) which is not marked for use with systemVMs.
As much as i like to have the system up running again, still the question
is why the reservation marked for usage of systemVMs - belonging to the
account system as the GUI says - isn't used anymore?
PS.:
I tried to create a new, additional ip range wich is marked as "Reserved
for SystemVMs" for the domain root. Afterwards I enabled the global setting
as described to "true".
Afterwards i distroyed a consoleproxy. The deployment of the new proxy
fails due to "insufficent address capacity". So it seems the IP Range is
not "respected"?
I checked the table "domain_vlan_map", which is used to get the vlan
information and there is NO entry of the newly created ip range...
regards
Christopher
Am Do., 1. Sept. 2022 um 01:01 Uhr schrieb Vash_X@gmx.de <Va...@gmx.de>:
> Hi Slavka,
>
> be prepared vor a longer answer as this seems to be a more specific
> problem - but first let me thank you for your guidance and support!
>
> > The discussion started about a problem migrating the system VMs, and
> probably many other issues will appear.
> Don't give up :-D I might see some light ahead....
>
> Bevore getting into the logs and my findings let me answer you your
> questions briefly.
> > Can you tell from which version you started the upgrade?
> I started from 4.16.1; the installation was builded with 4.15.1 and then
> updated to 4.16 / 4.16.1
> > Did the upgrade was successful for the management/agents?
> The upgrade was successfull - even if we had our issues which are
> currently in a ticket at github
> https://github.com/apache/cloudstack/issues/6680
> The upgrade of the management server was successfully performed too.
> > Did you restart the agents on all machines?
> We did as we had trouble with the changed detection of CPU speeds on the
> host.
> The whole "problem" of the SystemVM.iso is at least from my point of view
> no problem of the agent-update.
> a) we had the expection from reading the upgradeguide, that a "simple"
> patching of the systemVMs would be enough. That at least was our
> understanding of the "matrix" showing the patching capabilities of the
> different versions / "upgrade paths".
> b) currently we assume, that the main-reason for the failing migration is,
> that the SSVM / CPVM are still running without a clear "restart". Which we
> can't test as we still have to solve the problem of the public IP
> allocation.
> > Can you share the content in the directory
> `/usr/share/cloudstack-agent/lib/` from the hypervisor where the migration
> failed?
>
> activation-1.1.1.jar
> cloud-framework-events-4.17.0.1.jar
> error_prone_annotations-2.3.4.jar jaxb-impl-2.3.0.jar
> okio-2.8.0.jar
> annotations-13.0.jar
> cloud-framework-ipc-4.17.0.1.jar esapi-2.1.0.1.jar
> jbcrypt-1.0.0.jar
> opensaml-2.6.6.jar
> antisamy-1.5.3.jar
> cloud-framework-jobs-4.17.0.1.jar failureaccess-1.0.1.jar
> jcl-over-slf4j-1.7.5.jar
> openws-1.5.6.jar
> aopalliance-repackaged-2.5.0-b42.jar
> cloud-framework-managed-context-4.17.0.1.jar groovy-all-2.4.17.jar
> jersey-client-2.26.jar
> osgi-resource-locator-1.0.1.jar
> apache-log4j-extras-1.2.17.jar
> cloud-framework-rest-4.17.0.1.jar gson-1.7.2.jar
> jersey-common-2.26.jar
> protobuf-java-3.11.1.jar
> aspectjweaver-1.8.13.jar
> cloud-framework-security-4.17.0.1.jar guava-30.1-jre.jar
> jersey-entity-filtering-2.26.jar
> rados-0.6.0.jar
> aws-java-sdk-core-1.11.940.jar
> cloud-plugin-hypervisor-kvm-4.17.0.1.jar guava-testlib-18.0.jar
> jersey-hk2-2.26.jar
> reflections-0.9.12.jar
> aws-java-sdk-kms-1.11.940.jar
> cloud-plugin-network-ovs-4.17.0.1.jar hk2-api-2.5.0-b42.jar
> jersey-media-json-jackson-2.26.jar
> reload4j-1.2.20.jar
> aws-java-sdk-s3-1.11.940.jar
> cloud-plugin-storage-volume-scaleio-4.17.0.1.jar hk2-locator-2.5.0-b42.jar
> jersey-media-multipart-2.26.jar
> retrofit-2.9.0.jar
> batik-constants-1.14.jar
> cloud-secondary-storage-4.17.0.1.jar hk2-utils-2.5.0-b42.jar
> jmespath-java-1.11.940.jar
> rolling-maintenance
> batik-css-1.14.jar cloud-server-4.17.0.1.jar
> httpclient-4.5.13.jar
> jna-5.5.0.jar
> slf4j-api-1.7.5.jar
> batik-i18n-1.14.jar cloud-utils-4.17.0.1.jar
> httpcore-4.4.14.jar
> jna-platform-5.5.0.jar
> spring-aop-5.3.18.jar
> batik-shared-resources-1.14.jar commons-beanutils-1.9.2.jar
> influxdb-java-2.21.jar
> joda-time-2.10.9.jar
> spring-beans-5.3.18.jar
> batik-util-1.14.jar commons-codec-1.15.jar
> ini4j-0.5.4.jar
> jsch-0.1.55.jar
> spring-context-5.3.18.jar
> bcpkix-jdk15on-1.70.jar
> commons-collections-3.2.1.jar ion-java-1.0.2.jar
> json-20090211.jar
> spring-core-5.3.18.jar
> bcprov-jdk15on-1.70.jar commons-compress-1.21.jar
> j2objc-annotations-1.3.jar
> jsr305-1.3.9.jar
> spring-expression-5.3.18.jar
> bctls-jdk15on-1.70.jar
> commons-configuration-1.10.jar
> jackson-annotations-2.12.5.jar jul-to-slf4j-1.7.5.jar
> spring-jcl-5.3.18.jar
> bcutil-jdk15on-1.70.jar commons-daemon-1.2.3.jar
> jackson-core-2.12.5.jar
> kotlin-stdlib-1.4.10.jar
> spring-web-5.3.18.jar
> bsh-core-2.0b4.jar commons-dbcp2-2.8.0.jar
> jackson-databind-2.12.5.jar
> kotlin-stdlib-common-1.4.0.jar
> stax2-api-3.1.4.jar
> cglib-nodep-3.3.0.jar commons-digester-1.8.1.jar
> jackson-dataformat-cbor-2.6.7.jar
> kotlin-stdlib-jdk7-1.4.10.jar
> swagger-annotations-2.0.0.jar
> checker-qual-3.5.0.jar commons-email-1.5.jar
> jackson-datatype-jsr310-2.12.5.jar
> kotlin-stdlib-jdk8-1.4.10.jar
> tink-1.5.0.jar
> cloud-agent-4.17.0.1.jar commons-fileupload-1.4.jar
> jackson-jaxrs-base-2.12.1.jar
> libvirt-0.5.2.jar
> trilead-ssh2-build-217-jenkins-27.jar
> cloud-api-4.17.0.1.jar commons-httpclient-3.1.jar
> jackson-jaxrs-json-provider-2.12.1.jar
> libvirtqemuhook
> truth-0.23.jar
> cloud-core-4.17.0.1.jar commons-io-2.8.0.jar
> jackson-module-jaxb-annotations-2.8.4.jar
> listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
> velocity-1.7.jar
> cloud-engine-api-4.17.0.1.jar commons-lang-2.6.jar
> jasypt-1.9.3.jar
> log4j-over-slf4j-1.7.5.jar
> woodstox-core-5.0.3.jar
> cloud-engine-components-api-4.17.0.1.jar commons-lang3-3.11.jar
> java-ipv6-0.17.jar
> logging-interceptor-4.9.0.jar
> xalan-2.7.0.jar
> cloud-engine-schema-4.17.0.1.jar commons-logging-1.2.jar
> java-linstor-0.3.0.jar
> mail-1.5.0-b01.jar
> xercesImpl-2.8.0.jar
> cloud-engine-storage-4.17.0.1.jar commons-math3-3.6.1.jar
> javassist-3.27.0-GA.jar
> metrics-core-3.0.2.jar
> xmlgraphics-commons-2.6.jar
> cloud-engine-storage-configdrive-4.17.0.1.jar commons-net-3.7.2.jar
> javax.annotation-api-1.3.2.jar
> metrics-jvm-3.0.2.jar
> xmlpull-1.1.3.1.jar
> cloud-engine-storage-image-4.17.0.1.jar commons-pool2-2.9.0.jar
> javax.inject-1.jar
> mimepull-1.9.6.jar
> xmlschema-core-2.2.5.jar
> cloud-engine-storage-volume-4.17.0.1.jar commons-validator-1.6.jar
> javax.inject-2.5.0-b42.jar
> moshi-1.8.0.jar
> xmlsec-1.5.7.jar
> cloud-framework-agent-lb-4.17.0.1.jar converter-moshi-2.9.0.jar
> javax.mail-1.5.6.jar
> msgpack-core-0.8.21.jar
> xmltooling-1.4.6.jar
> cloud-framework-ca-4.17.0.1.jar cxf-core-3.2.14.jar
> javax.persistence-2.2.1.jar
> mxparser-1.2.2.jar
> xom-1.2.5.jar
> cloud-framework-cluster-4.17.0.1.jar
> cxf-rt-frontend-jaxrs-3.2.14.jar
> javax.servlet-api-4.0.1.jar nekohtml-1.9.16.jar
> xstream-1.4.19.jar
> cloud-framework-config-4.17.0.1.jar
> cxf-rt-transports-http-3.2.14.jar javax.ws.rs-api-2.1.jar
> netty-all-4.1.42.Final.jar
> cloud-framework-db-4.17.0.1.jar eddsa-0.3.0.jar
> jaxb-api-2.3.0.jar
> not-yet-commons-ssl-0.3.9.jar
> cloud-framework-direct-download-4.17.0.1.jar ehcache-core-2.6.11.jar
> jaxb-core-2.3.0.jar
>
> And now to the problem of the ip's....
> Thanks for your guidance, i guess i can give some more information. For
> the findings to understand i guess i need to elaborate a bit on the current
> configuration.
>
> Currently we are having only one Zone up and running (ID=1).
> In this Zone we have 3 different domains which are "Root","A","B".
> Each domain has a account linked: Root = Admin / A = AccA / B = AccB.
> The public network of the zone has currently 3 ip ranges which are: [Root]
> system (System VMs = true) / [Root] / [B]AccB (written like in the WebGUI
> which is i guess going to be important for understanding).
> This kind of configuration has been in place since the initial setup and
> has been working without problems.
>
> Afterwards you will find the logs with the database querys - i started
> with a deleting the stopped consoleproxy. i will only share the part when
> the process of configuring the NIC / IPs for the console proxy will take
> place.
> I put some statements into it - for me to get a better understanding of
> the process - and i hope that you will get through this a bit better. The
> comments are referenceing on the following "closing" statements. How ever i
> didn't deleted anything.
>
>
> ###################################################################### Nic
> allocation
>
> 2022-08-31 18:44:21,203 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Allocating nics for VM
> instance {id: "86", name: "v-86-VM", uuid:
> "6f77acb4-c65e-49e8-ae03-1175b3974412", type="ConsoleProxy"}
> 2022-08-31 18:44:21,203 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,204 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: start requested by:
> -Transaction.execute:39-NetworkOrchestrator.allocate:778-VirtualMachineManagerImpl$1.doInTransactionWithoutResult:478-TransactionCallbackWithExceptionNoReturn.doInTransaction:25-TransactionCallbackWithExceptionNoReturn.doInTransaction:21-Transaction.execute:40-VirtualMachineManagerImpl.allocate:469-VirtualMachineManagerImpl.allocate:528-ConsoleProxyManagerImpl.createProxyInstance:700-ConsoleProxyManagerImpl.startNew:572-ConsoleProxyManagerImpl.allocCapacity:809-ConsoleProxyManagerImpl.expandPool:1537
> 2022-08-31 18:44:21,204 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: has already been
> started.
> 2022-08-31 18:44:21,204 TRACE [o.a.c.e.o.NetworkOrchestrator]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) allocating networks for
> v-86-VM(template c20799e4-a990-449a-aae6-4b67374c7aff); 3 networks
> 2022-08-31 18:44:21,204 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,206 TRACE [c.c.u.d.T.Connection]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Creating a DB connection
> with txn: for 0: dbconn623913035. Stack:
> -TransactionLegacy.prepareStatement:466-TransactionLegacy.prepareAutoCloseStatement:459-GenericDaoBase.searchIncludingRemoved:396-GenericDaoBase.searchIncludingRemoved:359-GenericDaoBase.search:348-GenericDaoBase.search:1319-TemplateDeployAsIsDetailsDaoImpl.listDetailsByTemplateIdMatchingPrefix:64-TemplateDeployAsIsDetailsDaoImpl.listNetworkRequirementsByTemplateId:69-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62-DelegatingMethodAccessorImpl.invoke:43-Method.invoke:566
> 2022-08-31 18:44:21,206 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> template_deploy_as_is_details.id,
> template_deploy_as_is_details.template_id,
> template_deploy_as_is_details.name, template_deploy_as_is_details.value
> FROM template_deploy_as_is_details WHERE
> template_deploy_as_is_details.template_id = ? AND
> template_deploy_as_is_details.name LIKE ?
>
> #### template_deploy_as_is_details = empty --> no results returned
>
> 2022-08-31 18:44:21,206 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT
> template_deploy_as_is_details.id,
> template_deploy_as_is_details.template_id,
> template_deploy_as_is_details.name, template_deploy_as_is_details.value
> FROM template_deploy_as_is_details WHERE
> template_deploy_as_is_details.template_id = 213 AND
> template_deploy_as_is_details.name LIKE x'6E6574776F726B2D25'
> 2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Connection]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: Not closing DB
> connection because we're still in a transaction.
> 2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> networks.id, networks.mode, networks.broadcast_domain_type,
> networks.traffic_type, networks.name, networks.display_text,
> networks.broadcast_uri, networks.gateway, networks.cidr,
> networks.network_cidr, networks.network_offering_id, networks.vpc_id,
> networks.physical_network_id, networks.data_center_id, networks.related,
> networks.guru_name, networks.state, networks.redundant, networks.dns1,
> networks.domain_id, networks.account_id, networks.set_fields,
> networks.guru_data, networks.dns2, networks.network_domain,
> networks.removed, networks.created, networks.reservation_id, networks.uuid,
> networks.guest_type, networks.acl_type, networks.restart_required,
> networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr,
> networks.display_network, networks.network_acl_id, networks.streched_l2,
> networks.external_id FROM networks WHERE networks.id = ? AND
> networks.removed IS NULL
> 2022-08-31 18:44:21,209 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Allocating nic for vm VM
> instance {id: "86", name: "v-86-VM", uuid:
> "6f77acb4-c65e-49e8-ae03-1175b3974412", type="ConsoleProxy"} in network
> Network {"id": 200, "name": "null", "uuid":
> "1b0bf965-a391-478d-833d-3e37e4a15eb8", "networkofferingid": 1} with
> requested profile NicProfile
> {"broadcastUri":null,"iPv4Address":null,"id":0,"reservationId":null,"vmId":0}
> 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> data_center.id, data_center.name, data_center.description,
> data_center.dns1, data_center.dns2, data_center.ip6_dns1,
> data_center.ip6_dns2, data_center.internal_dns1, data_center.internal_dns2,
> data_center.router_mac_address, data_center.guest_network_cidr,
> data_center.domain_id, data_center.domain, data_center.networktype,
> data_center.dns_provider, data_center.dhcp_provider,
> data_center.gateway_provider, data_center.vpn_provider,
> data_center.userdata_provider, data_center.lb_provider,
> data_center.firewall_provider, data_center.mac_address,
> data_center.zone_token, data_center.removed, data_center.allocation_state,
> data_center.uuid, data_center.is_security_group_enabled,
> data_center.is_local_storage_enabled, data_center.sort_key FROM data_center
> WHERE data_center.id = ? AND data_center.removed IS NULL
>
> ### Allocating public nic
>
> 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT networks.id,
> networks.mode, networks.broadcast_domain_type, networks.traffic_type,
> networks.name, networks.display_text, networks.broadcast_uri,
> networks.gateway, networks.cidr, networks.network_cidr,
> networks.network_offering_id, networks.vpc_id,
> networks.physical_network_id, networks.data_center_id, networks.related,
> networks.guru_name, networks.state, networks.redundant, networks.dns1,
> networks.domain_id, networks.account_id, networks.set_fields,
> networks.guru_data, networks.dns2, networks.network_domain,
> networks.removed, networks.created, networks.reservation_id, networks.uuid,
> networks.guest_type, networks.acl_type, networks.restart_required,
> networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr,
> networks.display_network, networks.network_acl_id, networks.streched_l2,
> networks.external_id FROM networks WHERE networks.id = 200 AND
> networks.removed IS NULL
> 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> account.id, account.account_name, account.type, account.role_id,
> account.domain_id, account.state, account.created, account.removed,
> account.cleanup_needed, account.network_domain, account.uuid,
> account.default_zone_id, account.default FROM account WHERE account.id =
> ? AND account.removed IS NULL
>
> ### Getting Zone Information
>
> 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT data_center.id,
> data_center.name, data_center.description, data_center.dns1,
> data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2,
> data_center.internal_dns1, data_center.internal_dns2,
> data_center.router_mac_address, data_center.guest_network_cidr,
> data_center.domain_id, data_center.domain, data_center.networktype,
> data_center.dns_provider, data_center.dhcp_provider,
> data_center.gateway_provider, data_center.vpn_provider,
> data_center.userdata_provider, data_center.lb_provider,
> data_center.firewall_provider, data_center.mac_address,
> data_center.zone_token, data_center.removed, data_center.allocation_state,
> data_center.uuid, data_center.is_security_group_enabled,
> data_center.is_local_storage_enabled, data_center.sort_key FROM data_center
> WHERE data_center.id = 1 AND data_center.removed IS NULL
> 2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: start requested by:
> -Transaction.execute:39-IpAddressManagerImpl.listAvailablePublicIps:824-IpAddressManagerImpl.fetchNewPublicIp:811-IpAddressManagerImpl.fetchNewPublicIp:803-IpAddressManagerImpl.assignPublicIpAddress:784-PublicNetworkGuru.getIp:125-PublicNetworkGuru.allocate:172-NetworkOrchestrator.allocateNic:973-NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice:823-NetworkOrchestrator$3.doInTransactionWithoutResult:797-TransactionCallbackWithExceptionNoReturn.doInTransaction:25-TransactionCallbackWithExceptionNoReturn.doInTransaction:21
> 2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: has already been
> started.
> 2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> data_center.id, data_center.name, data_center.description,
> data_center.dns1, data_center.dns2, data_center.ip6_dns1,
> data_center.ip6_dns2, data_center.internal_dns1, data_center.internal_dns2,
> data_center.router_mac_address, data_center.guest_network_cidr,
> data_center.domain_id, data_center.domain, data_center.networktype,
> data_center.dns_provider, data_center.dhcp_provider,
> data_center.gateway_provider, data_center.vpn_provider,
> data_center.userdata_provider, data_center.lb_provider,
> data_center.firewall_provider, data_center.mac_address,
> data_center.zone_token, data_center.removed, data_center.allocation_state,
> data_center.uuid, data_center.is_security_group_enabled,
> data_center.is_local_storage_enabled, data_center.sort_key FROM data_center
> WHERE data_center.id = ? AND data_center.removed IS NULL
>
> ### Account Information "system"
>
> 2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT account.id,
> account.account_name, account.type, account.role_id, account.domain_id,
> account.state, account.created, account.removed, account.cleanup_needed,
> account.network_domain, account.uuid, account.default_zone_id,
> account.default FROM account WHERE account.id = 1 AND account.removed IS
> NULL
> 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> networks.id, networks.mode, networks.broadcast_domain_type,
> networks.traffic_type, networks.name, networks.display_text,
> networks.broadcast_uri, networks.gateway, networks.cidr,
> networks.network_cidr, networks.network_offering_id, networks.vpc_id,
> networks.physical_network_id, networks.data_center_id, networks.related,
> networks.guru_name, networks.state, networks.redundant, networks.dns1,
> networks.domain_id, networks.account_id, networks.set_fields,
> networks.guru_data, networks.dns2, networks.network_domain,
> networks.removed, networks.created, networks.reservation_id, networks.uuid,
> networks.guest_type, networks.acl_type, networks.restart_required,
> networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr,
> networks.display_network, networks.network_acl_id, networks.streched_l2,
> networks.external_id FROM networks WHERE networks.id = ? AND
> networks.removed IS NULL
>
> ### Getting Zone Information
>
> 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT data_center.id,
> data_center.name, data_center.description, data_center.dns1,
> data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2,
> data_center.internal_dns1, data_center.internal_dns2,
> data_center.router_mac_address, data_center.guest_network_cidr,
> data_center.domain_id, data_center.domain, data_center.networktype,
> data_center.dns_provider, data_center.dhcp_provider,
> data_center.gateway_provider, data_center.vpn_provider,
> data_center.userdata_provider, data_center.lb_provider,
> data_center.firewall_provider, data_center.mac_address,
> data_center.zone_token, data_center.removed, data_center.allocation_state,
> data_center.uuid, data_center.is_security_group_enabled,
> data_center.is_local_storage_enabled, data_center.sort_key FROM data_center
> WHERE data_center.id = 1 AND data_center.removed IS NULL
> 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> account_vlan_map.id, account_vlan_map.account_id,
> account_vlan_map.vlan_db_id FROM account_vlan_map WHERE
> account_vlan_map.account_id = ?
>
> ### Getting network information
>
> 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT networks.id,
> networks.mode, networks.broadcast_domain_type, networks.traffic_type,
> networks.name, networks.display_text, networks.broadcast_uri,
> networks.gateway, networks.cidr, networks.network_cidr,
> networks.network_offering_id, networks.vpc_id,
> networks.physical_network_id, networks.data_center_id, networks.related,
> networks.guru_name, networks.state, networks.redundant, networks.dns1,
> networks.domain_id, networks.account_id, networks.set_fields,
> networks.guru_data, networks.dns2, networks.network_domain,
> networks.removed, networks.created, networks.reservation_id, networks.uuid,
> networks.guest_type, networks.acl_type, networks.restart_required,
> networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr,
> networks.display_network, networks.network_acl_id, networks.streched_l2,
> networks.external_id FROM networks WHERE networks.id = null AND
> networks.removed IS NULL
> 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> domain_vlan_map.id, domain_vlan_map.domain_id, domain_vlan_map.vlan_db_id
> FROM domain_vlan_map WHERE domain_vlan_map.domain_id = ?
>
> #### Account_vlan_map - no entry for account "System" nor for Account
> "Admin"
>
> 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT account_vlan_map.id,
> account_vlan_map.account_id, account_vlan_map.vlan_db_id FROM
> account_vlan_map WHERE account_vlan_map.account_id = 1
> 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT vlan.id,
> vlan.vlan_id, vlan.vlan_gateway, vlan.vlan_netmask, vlan.ip6_gateway,
> vlan.ip6_cidr, vlan.data_center_id, vlan.ip4_range, vlan.ip6_range,
> vlan.network_id, vlan.physical_network_id, vlan.vlan_type, vlan.uuid,
> vlan.removed, vlan.created FROM vlan LEFT OUTER JOIN domain_vlan_map ON
> vlan.id=domain_vlan_map.vlan_db_id LEFT OUTER JOIN account_vlan_map ON
> vlan.id=account_vlan_map.vlan_db_id WHERE vlan.data_center_id = ? AND
> vlan.removed IS NULL AND (domain_vlan_map.domain_id IS NULL ) AND
> (account_vlan_map.account_id IS NULL )
>
> ### Checking for availeable VLANS wich are NOT assigned to ANY Account NOR
> to ANY Domain --> Result is the intended vlan with expected IP Range
>
> 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT domain_vlan_map.id,
> domain_vlan_map.domain_id, domain_vlan_map.vlan_db_id FROM domain_vlan_map
> WHERE domain_vlan_map.domain_id = 1
> 2022-08-31 18:44:21,214 TRACE [c.c.u.d.GenericDaoBase]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) join search statement is
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT vlan.id, vlan.vlan_id,
> vlan.vlan_gateway, vlan.vlan_netmask, vlan.ip6_gateway, vlan.ip6_cidr,
> vlan.data_center_id, vlan.ip4_range, vlan.ip6_range, vlan.network_id,
> vlan.physical_network_id, vlan.vlan_type, vlan.uuid, vlan.removed,
> vlan.created FROM vlan LEFT OUTER JOIN domain_vlan_map ON vlan.id=domain_vlan_map.vlan_db_id
> LEFT OUTER JOIN account_vlan_map ON vlan.id=account_vlan_map.vlan_db_id
> WHERE vlan.data_center_id = 1 AND vlan.removed IS NULL AND
> (domain_vlan_map.domain_id IS NULL ) AND (account_vlan_map.account_id IS
> NULL )
> 2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> configuration.instance, configuration.component, configuration.name,
> configuration.value, configuration.default_value,
> configuration.description, configuration.category,
> configuration.is_dynamic, configuration.scope, configuration.updated FROM
> configuration WHERE configuration.name = ?
>
> ### Getting further information on the vlan --> correct choice of expected
> VLAN + IP-Range
>
> 2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT vlan.id, vlan.vlan_id,
> vlan.vlan_gateway, vlan.vlan_netmask, vlan.ip6_gateway, vlan.ip6_cidr,
> vlan.data_center_id, vlan.ip4_range, vlan.ip6_range, vlan.network_id,
> vlan.physical_network_id, vlan.vlan_type, vlan.uuid, vlan.removed,
> vlan.created FROM vlan LEFT OUTER JOIN domain_vlan_map ON vlan.id=domain_vlan_map.vlan_db_id
> LEFT OUTER JOIN account_vlan_map ON vlan.id=account_vlan_map.vlan_db_id
> WHERE vlan.data_center_id = 1 AND vlan.removed IS NULL AND
> (domain_vlan_map.domain_id IS NULL ) AND (account_vlan_map.account_id IS
> NULL )
> 2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> user_ip_address.id, user_ip_address.account_id,
> user_ip_address.domain_id, user_ip_address.public_ip_address,
> user_ip_address.data_center_id, user_ip_address.source_nat,
> user_ip_address.allocated, user_ip_address.vlan_db_id,
> user_ip_address.one_to_one_nat, user_ip_address.vm_id,
> user_ip_address.state, user_ip_address.mac_address,
> user_ip_address.source_network_id, user_ip_address.network_id,
> user_ip_address.uuid, user_ip_address.physical_network_id,
> user_ip_address.is_system, user_ip_address.vpc_id,
> user_ip_address.dnat_vmip, user_ip_address.is_portable,
> user_ip_address.display, user_ip_address.rule_state,
> user_ip_address.forsystemvms, user_ip_address.removed,
> user_ip_address.created FROM user_ip_address INNER JOIN vlan ON
> user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id =
> ? AND user_ip_address.allocated IS NULL AND user_ip_address.vlan_db_id IN
> (?,?,?,?) AND user_ip_address.forsystemvms = ? AND
> user_ip_address.removed IS NULL AND (vlan.vlan_type = ? ) ORDER BY
> user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id ASC LIMIT
> 0, 1 FOR UPDATE
>
> ### Check of setting system.vm.public.ip.reservation.mode.strictness -->
> value "true"
>
> 2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT configuration.instance,
> configuration.component, configuration.name, configuration.value,
> configuration.default_value, configuration.description,
> configuration.category, configuration.is_dynamic, configuration.scope,
> configuration.updated FROM configuration WHERE configuration.name =
> x'73797374656D2E766D2E7075626C69632E69702E7265736572766174696F6E2E6D6F64652E7374726963746E657373'
>
> 2022-08-31 18:44:21,215 TRACE [c.c.u.d.GenericDaoBase]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) join search statement is
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT user_ip_address.id,
> user_ip_address.account_id, user_ip_address.domain_id,
> user_ip_address.public_ip_address, user_ip_address.data_center_id,
> user_ip_address.source_nat, user_ip_address.allocated,
> user_ip_address.vlan_db_id, user_ip_address.one_to_one_nat,
> user_ip_address.vm_id, user_ip_address.state, user_ip_address.mac_address,
> user_ip_address.source_network_id, user_ip_address.network_id,
> user_ip_address.uuid, user_ip_address.physical_network_id,
> user_ip_address.is_system, user_ip_address.vpc_id,
> user_ip_address.dnat_vmip, user_ip_address.is_portable,
> user_ip_address.display, user_ip_address.rule_state,
> user_ip_address.forsystemvms, user_ip_address.removed,
> user_ip_address.created FROM user_ip_address INNER JOIN vlan ON
> user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id =
> 1 AND user_ip_address.allocated IS NULL AND user_ip_address.vlan_db_id IN
> (3,10,13,14) AND user_ip_address.forsystemvms = 1 AND
> user_ip_address.removed IS NULL AND (vlan.vlan_type = 'VirtualNetwork' )
> ORDER BY user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id ASC
> LIMIT 0, 1 FOR UPDATE
> 2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> account_details.id, account_details.account_id, account_details.name,
> account_details.value FROM account_details WHERE account_details.account_id
> = ? AND account_details.name = ? ORDER BY RAND() LIMIT 1
>
> ### Trying to fetch IP-Address: No value / ip address returned - Domain
> "Root" shall provide ip addresses; thats where the vlanIDs are comming from
> --> user_ip_address.vlan_db_id IN (3,10,13,14)
>
> 2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT user_ip_address.id,
> user_ip_address.account_id, user_ip_address.domain_id,
> user_ip_address.public_ip_address, user_ip_address.data_center_id,
> user_ip_address.source_nat, user_ip_address.allocated,
> user_ip_address.vlan_db_id, user_ip_address.one_to_one_nat,
> user_ip_address.vm_id, user_ip_address.state, user_ip_address.mac_address,
> user_ip_address.source_network_id, user_ip_address.network_id,
> user_ip_address.uuid, user_ip_address.physical_network_id,
> user_ip_address.is_system, user_ip_address.vpc_id,
> user_ip_address.dnat_vmip, user_ip_address.is_portable,
> user_ip_address.display, user_ip_address.rule_state,
> user_ip_address.forsystemvms, user_ip_address.removed,
> user_ip_address.created FROM user_ip_address INNER JOIN vlan ON
> user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id =
> 1 AND user_ip_address.allocated IS NULL AND user_ip_address.vlan_db_id IN
> (3,10,13,14) AND user_ip_address.forsystemvms = 1 AND
> user_ip_address.removed IS NULL AND (vlan.vlan_type = 'VirtualNetwork' )
> ORDER BY user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id ASC
> LIMIT 0, 1 FOR UPDATE
> 2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> configuration.instance, configuration.component, configuration.name,
> configuration.value, configuration.default_value,
> configuration.description, configuration.category,
> configuration.is_dynamic, configuration.scope, configuration.updated FROM
> configuration WHERE configuration.name = ? ORDER BY RAND() LIMIT 1
>
> ### Trying to get information from the table account_detail for the
> account "system". Table is empty so no result back
>
> 2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT account_details.id,
> account_details.account_id, account_details.name, account_details.value
> FROM account_details WHERE account_details.account_id = 1 AND
> account_details.name = x'7573652E73797374656D2E7075626C69632E697073'
> ORDER BY RAND() LIMIT 1
> 2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> configuration.instance, configuration.component, configuration.name,
> configuration.value, configuration.default_value,
> configuration.description, configuration.category,
> configuration.is_dynamic, configuration.scope, configuration.updated FROM
> configuration WHERE configuration.name = ?
>
> ### Check for Configuration enable.account.settings.for.domain --> current
> value = false
>
> 2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT configuration.instance,
> configuration.component, configuration.name, configuration.value,
> configuration.default_value, configuration.description,
> configuration.category, configuration.is_dynamic, configuration.scope,
> configuration.updated FROM configuration WHERE configuration.name =
> x'656E61626C652E6163636F756E742E73657474696E67732E666F722E646F6D61696E'
> ORDER BY RAND() LIMIT 1
> 2022-08-31 18:44:21,217 WARN [c.c.n.IpAddressManagerImpl]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Unable to get ip address in
> zone id=1, vlanId id=[3, 10, 13, 14]
> 2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> data_center.id, data_center.name, data_center.description,
> data_center.dns1, data_center.dns2, data_center.ip6_dns1,
> data_center.ip6_dns2, data_center.internal_dns1, data_center.internal_dns2,
> data_center.router_mac_address, data_center.guest_network_cidr,
> data_center.domain_id, data_center.domain, data_center.networktype,
> data_center.dns_provider, data_center.dhcp_provider,
> data_center.gateway_provider, data_center.vpn_provider,
> data_center.userdata_provider, data_center.lb_provider,
> data_center.firewall_provider, data_center.mac_address,
> data_center.zone_token, data_center.removed, data_center.allocation_state,
> data_center.uuid, data_center.is_security_group_enabled,
> data_center.is_local_storage_enabled, data_center.sort_key FROM data_center
> WHERE data_center.id = ? AND data_center.removed IS NULL
>
> ### Check for setting use.system.public.ips --> current value = false
>
> 2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT configuration.instance,
> configuration.component, configuration.name, configuration.value,
> configuration.default_value, configuration.description,
> configuration.category, configuration.is_dynamic, configuration.scope,
> configuration.updated FROM configuration WHERE configuration.name =
> x'7573652E73797374656D2E7075626C69632E697073'
>
> ### getting information for the current data_center / zone
>
> 2022-08-31 18:44:21,219 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT data_center.id,
> data_center.name, data_center.description, data_center.dns1,
> data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2,
> data_center.internal_dns1, data_center.internal_dns2,
> data_center.router_mac_address, data_center.guest_network_cidr,
> data_center.domain_id, data_center.domain, data_center.networktype,
> data_center.dns_provider, data_center.dhcp_provider,
> data_center.gateway_provider, data_center.vpn_provider,
> data_center.userdata_provider, data_center.lb_provider,
> data_center.firewall_provider, data_center.mac_address,
> data_center.zone_token, data_center.removed, data_center.allocation_state,
> data_center.uuid, data_center.is_security_group_enabled,
> data_center.is_local_storage_enabled, data_center.sort_key FROM data_center
> WHERE data_center.id = 1 AND data_center.removed IS NULL
> 2022-08-31 18:44:21,219 DEBUG [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Rolling back the
> transaction: Time = 17 Name = consoleproxy-1; called by
> -TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-Transaction.execute:38-IpAddressManagerImpl.listAvailablePublicIps:824-IpAddressManagerImpl.fetchNewPublicIp:811-IpAddressManagerImpl.fetchNewPublicIp:803-IpAddressManagerImpl.assignPublicIpAddress:784-PublicNetworkGuru.getIp:125-PublicNetworkGuru.allocate:172-NetworkOrchestrator.allocateNic:973-NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice:823
> 2022-08-31 18:44:21,219 TRACE [c.c.u.d.T.Lock]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) SQL
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT user_ip_address.id,
> user_ip_address.account_id, user_ip_address.domain_id,
> user_ip_address.public_ip_address, user_ip_address.data_center_id,
> user_ip_address.source_nat, user_ip_address.allocated,
> user_ip_address.vlan_db_id, user_ip_address.one_to_one_nat,
> user_ip_address.vm_id, user_ip_address.state, user_ip_address.mac_address,
> user_ip_address.source_network_id, user_ip_address.network_id,
> user_ip_address.uuid, user_ip_address.physical_network_id,
> user_ip_address.is_system, user_ip_address.vpc_id,
> user_ip_address.dnat_vmip, user_ip_address.is_portable,
> user_ip_address.display, user_ip_address.rule_state,
> user_ip_address.forsystemvms, user_ip_address.removed,
> user_ip_address.created FROM user_ip_address INNER JOIN vlan ON
> user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id =
> 1 AND user_ip_address.allocated IS NULL AND user_ip_address.vlan_db_id IN
> (3,10,13,14) AND user_ip_address.forsystemvms = 1 AND
> user_ip_address.removed IS NULL AND (vlan.vlan_type = 'VirtualNetwork' )
> ORDER BY user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id ASC
> LIMIT 0, 1 FOR UPDATE took 4
> 2022-08-31 18:44:21,219 TRACE [c.c.u.d.T.Connection]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing DB connection:
> dbconn623913035
> 2022-08-31 18:44:21,220 WARN [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Unable to allocate proxy
> [Console VM instance {id: "86", name: "v-86-VM", uuid:
> "6f77acb4-c65e-49e8-ae03-1175b3974412", type="ConsoleProxy"}] on zone [1]
> due to [Insufficient address capacity].
> com.cloud.exception.InsufficientAddressCapacityException: Insufficient
> address capacityScope=interface com.cloud.dc.DataCenter; id=1
>
> following the settings of our global config which are mentioned in the
> above log:
> *system.vm.public.ip.reservation.mode.strictness* - true
> *use.system.public.ips* - true
> *enable.account.settings.for.domain* - false
>
> The challange now is to get all the things together into a context...
> From my impression changeing the value for 'use.system.public.ips' may not
> be helping, as at this point the "warning" already has been dropped. But
> would only be a "workaround".
>
> Questions from my side:
> - Looking into the web GUI for the IP-Ranges for the public network i am
> getting quite curios. The current entry for '[Root] system' is strange.
> Clicking on it i am getting a view stateing that this range shall be
> "connected" to the Account 'System', the Domain 'Root' and SystemVMs 'true'.
> While the 'System VMs = true' value can be found in the database, the vlan
> / ip - range isn't connected to a account or domain at all in the database.
> - What is the overall 'effect' of setting *system.vm.public.ip.reservation.mode.strictness?
> *From the description for me a "true" will prevent the usage of of public
> IPs from the system pool. But what is defined as "system Pool"? Looking at
> the GUI I would assume that the "account" System has a dedicated pool of
> public IPs for systemVMs (speaking that this would fail as there is no
> reference in the DB for a mapping of any Pool / Range for the account
> 'system').
>
> To summerize this seems more like a "unlucky" configuration or missing
> db.entry / bug throughout versions. Or a mix of both.
> How ever i don't really understand why there is a query to find VLANs not
> attached to a Account or an Domain - or is this the SystemPool?
>
> Thanks again for all the help!
> Chris
>
>
>
>
>
> Am Mi., 31. Aug. 2022 um 10:52 Uhr schrieb Slavka Peleva <
> slavkap@storpool.com>:
>
>> Hi Chris,
>>
>> According to the logs you shared and looking at the code, I guess CS gets
>> the VLAN IDs by domain id, not by zone id. But to be more precise in this,
>> can you enable the TRACE level in your logs and share the exact query that
>> CS executes?
>> If I'm right with this, you can enable *just for now* the setting
>> `use.system.public.ips` (To redeploy the system VMs). This should pick
>> VLANs by zone ID. I hope that the deployment won't fail this time! If the
>> system VMs are recreated, can you disable the setting
>> `use.system.public.ips` (because I don't know if this will affect something
>> else)?
>>
>> The discussion started about a problem migrating the system VMs, and
>> probably many other issues will appear.
>> Can you tell from which version you started the upgrade? (two weeks ago,
>> I upgraded one of my environments from 4.11.3.0 to the main branch
>> 4.18.0.0-SNAPSHOT without any issues)
>> Did the upgrade was successful for the management/agents?
>> Did you restart the agents on all machines? My guess is the new code
>> isn't loaded. It's strange that CS is looking for systemvm.iso when 4.17
>> doesn't know about it.
>> Can you share the content in the directory
>> `/usr/share/cloudstack-agent/lib/` from the hypervisor where the migration
>> failed?
>>
>> Best regards,
>> Slavka
>>
>>
>>
>
> On Wed, Aug 31, 2022 at 2:35 AM Vash_X@gmx.de <Va...@gmx.de> wrote:
>>
>>> Hi Nicolas,
>>>
>>> > What is the value for the setting
>>> ‘system.vm.public.ip.reservation.mode.strictness’?
>>> This is set to 'true'.
>>>
>>> > Can you please check if this query gets any result?
>>> Yes the query gives me results - all availeable "public" ip addresses of
>>> the reserved range for the systemVMs
>>>
>>> # public_ip_address vlan_db_id vlan_id
>>> 1 10.129.0.10 9 1000
>>> 2 10.129.0.11 9 1000
>>> 3 10.129.0.12 9 1000
>>> 4 10.129.0.13 9 1000
>>> 5 10.129.0.14 9 1000
>>> 6 10.129.0.15 9 1000
>>> 7 10.129.0.16 9 1000
>>> 8 10.129.0.18 9 1000
>>> 9 10.129.0.19 9 1000
>>> Regarding the vlanId id=[3, 10, 13, 14]
>>> I looked up my table "vlan" and am quite shocked:
>>>
>>> the ids 3, 13, 14 have a value for "range removed", so i don't know why
>>> the
>>> console proxy is looking for these networks.
>>> the id "10" is is still active - however the ipv4 range is not "the right
>>> one". That is assigned as the public ip-range for the root-account but
>>> not
>>> for use of systemvms...
>>>
>>> Am Di., 30. Aug. 2022 um 16:43 Uhr schrieb Nicolas Vazquez <
>>> Nicolas.Vazquez@shapeblue.com>:
>>>
>>> > Hi Chris,
>>> >
>>> > What is the value for the setting
>>> > ‘system.vm.public.ip.reservation.mode.strictness’?
>>> >
>>> > Looks like the error is not being able to get an IP address for zone
>>> id=1,
>>> > vlanId id=[3, 10, 13, 14].
>>> > If I understood correctly you have defined a range for system VMs? Can
>>> you
>>> > please check if this query gets any result?
>>> >
>>> > select i.public_ip_address, i.vlan_db_id, v.vlan_id from
>>> user_ip_address i
>>> > join vlan v on (i.vlan_db_id = v.id)
>>> > where i.forsystemvms = 1 and i.allocated is null and i.state = 'Free'
>>> and
>>> > v.removed is null and i.data_center_id = 1;
>>> >
>>> > Regards,
>>> > Nicolas Vazquez
>>> >
>>> >
>>> > From: Vash_X@gmx.de <Va...@gmx.de>
>>> > Date: Tuesday, 30 August 2022 at 09:38
>>> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>, Slavka
>>> > Peleva <sl...@storpool.com>
>>> > Subject: Re: Error migrate System VMs after upgrageing to 4.17
>>> > Hi Slavka,
>>> >
>>> > that was my first thought, too. I checked beforehand and confirmed with
>>> > your provided SQL Statements:
>>> > *`select * from user_ip_address where forsystemvms = 1;`* <----
>>> returns my
>>> > defined (Public) IPs out of the correct range for SystemVMs
>>> > *`SELECT * FROM clou*d.user_ip_address where allocated IS NULL AND
>>> removed
>>> > IS NULL; <-- All IPs shown beside the one which is currently in use
>>> for the
>>> > running ssvm. Meaning from my point of view "free" for allocation for
>>> other
>>> > system VMs.
>>> >
>>> > Querys performed directly on the DB-Server.
>>> > I can't see a reason why there shouldn't be any IPs left for
>>> allocation.
>>> >
>>> > Also it is interesting why the "reasoning" for the failing deployment
>>> > changes from "no IP availeable" to "no useable volumes found".
>>> > How ever i can't see anything missing at the moment, what makes me even
>>> > more puzzeled.
>>> >
>>> > Regards
>>> > Chris
>>> >
>>> > Am Di., 30. Aug. 2022 um 13:31 Uhr schrieb Slavka Peleva
>>> > <sl...@storpool.com.invalid>:
>>> >
>>> > > Hi Chris,
>>> > >
>>> > > The deployment fails because CS cannot find a free IP address to
>>> allocate
>>> > > for the system VM. Can you check if you explicitly added a range for
>>> the
>>> > > system VMs, or you can check with this `select * from user_ip_address
>>> > where
>>> > > forsystemvms = 1;` If you don't have any ranges, you can get free IPs
>>> > > with `SELECT
>>> > > * FROM cloud.user_ip_address where allocated IS NULL AND removed IS
>>> > NULL;`
>>> > > . To get the exact query executed by CS, you need to increase the log
>>> > level
>>> > > to TRACE.
>>> > >
>>> > > Best regards,
>>> > > Slavka
>>> > >
>>> >
>>> >
>>> >
>>> > > On Mon, Aug 29, 2022 at 8:03 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
>>> > >
>>> > > > Hi Slavka,
>>> > > >
>>> > > > no, i am not using any kind of custome offerings.
>>> > > >
>>> > > > I try to provide a more detailled log this time.
>>> > > > Situation:
>>> > > > v-83-VM was the "broken" consoleproxy. So the following log starts
>>> with
>>> > > > it's expunging.
>>> > > > Afterwards the new consoleproxy shall be deployed - v-85-VM.
>>> > > >
>>> > > > This first deployment (logid:dcfe0999) fails due to Insufficent
>>> address
>>> > > > capacity - which i can't really understand or see in the database,
>>> > > checking
>>> > > > the availeable IPs for systemVMs (given that i might looking up the
>>> > wrong
>>> > > > tables....).
>>> > > > How ever CS is working as expected trying to deploy the console
>>> proxy
>>> > > > (logid:440fd101) Sync job-4288).
>>> > > > That job fails with "Unable to create deployment, no usable volumes
>>> > found
>>> > > > for the VM: 85".
>>> > > > Afterwards i have another try to start the consoleproxy (job-4289)
>>> > which
>>> > > > fails for the same reason. And thats how things go on in the log
>>> > file...
>>> > > >
>>> > > >
>>> > > > 2022-08-29 17:01:16,224 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>>> > (logid:5fe8559a)
>>> > > > Cleaning up hypervisor data structures (ex. SRs in XenServer) for
>>> > managed
>>> > > > storage. Data from VM instance {id: "83", name: "v-83-VM", uuid:
>>> > > > "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}.
>>> > > > 2022-08-29 17:01:16,225 DEBUG [o.a.c.e.o.VolumeOrchestrator]
>>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>>> > (logid:5fe8559a)
>>> > > > Cleaning storage for vm: 83
>>> > > > 2022-08-29 17:01:16,233 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>>> > (logid:5fe8559a)
>>> > > > Expunged VM instance {id: "83", name: "v-83-VM", uuid:
>>> > > > "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
>>> > > > 2022-08-29 17:01:16,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>>> > (logid:5fe8559a)
>>> > > > Complete async job-4286, jobStatus: SUCCEEDED, resultCode: 0,
>>> result:
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.api.response.SystemVmResponse/systemvm/{"id":"b1d1f9c6-eafe-4527-9335-e649c646aab0","systemvmtype":"consoleproxy","zoneid":"48e2e928-3300-43b5-8e3a-d9b358022502","zonename":"xxxxxxxxxx","dns1":"10.129.0.4","dns2":"195.10.195.195","name":"v-83-VM","templateid":"c20799e4-a990-449a-aae6-4b67374c7aff","templatename":"systemvm-kvm-4.17.0","created":"2022-08-29T10:38:26+0200","state":"Stopped","isdynamicallyscalable":"false","hasannotations":"false"}
>>> > > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>>> > (logid:5fe8559a)
>>> > > > Publish async job-4286 complete on message bus
>>> > > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>>> > (logid:5fe8559a)
>>> > > > Wake up jobs related to job-4286
>>> > > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>>> > (logid:5fe8559a)
>>> > > > Update db status for job-4286
>>> > > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>>> > (logid:5fe8559a)
>>> > > > Wake up jobs joined with job-4286 and disjoin all subjobs created
>>> from
>>> > > job-
>>> > > > 4286
>>> > > > 2022-08-29 17:01:16,260 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286) (logid:5fe8559a) Done
>>> > > executing
>>> > > >
>>> org.apache.cloudstack.api.command.admin.systemvm.DestroySystemVmCmd for
>>> > > > job-4286
>>> > > > 2022-08-29 17:01:16,260 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286) (logid:5fe8559a) Remove
>>> > > > job-4286 from job monitoring
>>> > > > 2022-08-29 17:01:17,691 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-6:ctx-6af18142) (logid:fe263a70) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > >
>>> > > > 2022-08-29 17:01:21,694 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-6:ctx-89b1442f) (logid:0efc40b2) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:01:25,698 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-5:ctx-62313d05) (logid:7073663f) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:01:25,767 DEBUG [c.c.s.StatsCollector]
>>> > > > (secstorage-1:ctx-c80e5a6e) (logid:f8e3eb60) Verifying image
>>> storage
>>> > [1].
>>> > > > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
>>> > > > 2022-08-29 17:01:25,773 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
>>> > > > (secstorage-1:ctx-c80e5a6e) (logid:f8e3eb60) Zone [1] is ready to
>>> > launch
>>> > > > secondary storage VM.
>>> > > > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Zone 1 is ready to
>>> > launch
>>> > > > console proxy
>>> > > > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Expand console proxy
>>> > > standby
>>> > > > capacity for zone xxxxxxxxxx
>>> > > > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating console
>>> proxy
>>> > > > standby capacity for zone [1].
>>> > > > 2022-08-29 17:01:26,032 INFO [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) No stopped console
>>> proxy
>>> > > is
>>> > > > available, need to allocate a new console proxy
>>> > > > 2022-08-29 17:01:26,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Assign console proxy
>>> > from
>>> > > a
>>> > > > newly started instance for request from data center : 1
>>> > > > 2022-08-29 17:01:26,056 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing
>>> network
>>> > > > configuration for offering [Network Offering
>>> > > > [1-Public-System-Public-Network]: Network {"id": 200, "name":
>>> "null",
>>> > > > "uuid": "1b0bf965-a391-478d-833d-3e37e4a15eb8",
>>> "networkofferingid": 1}
>>> > > > 2022-08-29 17:01:26,056 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
>>> > > > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account
>>> {"id": 1,
>>> > > > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
>>> > > > 2022-08-29 17:01:26,064 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing
>>> network
>>> > > > configuration for offering [Network Offering
>>> > > > [3-Control-System-Control-Network]: Network {"id": 202, "name":
>>> "null",
>>> > > > "uuid": "2af42518-fbf4-4a35-9d6b-370e6ad315c7",
>>> "networkofferingid": 3}
>>> > > > 2022-08-29 17:01:26,064 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
>>> > > > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account
>>> {"id": 1,
>>> > > > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
>>> > > > 2022-08-29 17:01:26,069 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing
>>> network
>>> > > > configuration for offering [Network Offering
>>> > > > [2-Management-System-Management-Network]: Network {"id": 201,
>>> "name":
>>> > > > "null", "uuid": "53711d87-a3c6-4f33-bfce-c30e0de72760",
>>> > > > "networkofferingid": 2}
>>> > > > 2022-08-29 17:01:26,069 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
>>> > > > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account
>>> {"id": 1,
>>> > > > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
>>> > > > 2022-08-29 17:01:26,077 INFO [c.c.v.VirtualMachineManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) allocating virtual
>>> > machine
>>> > > > from template:c20799e4-a990-449a-aae6-4b67374c7aff with
>>> > hostname:v-85-VM
>>> > > > and 3 networks
>>> > > > 2022-08-29 17:01:26,079 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating entries
>>> for
>>> > VM:
>>> > > > VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
>>> > > > 2022-08-29 17:01:26,082 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating nics for
>>> VM
>>> > > > instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
>>> > > > 2022-08-29 17:01:26,083 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating nic for
>>> vm VM
>>> > > > instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} in
>>> network
>>> > > > Network {"id": 200, "name": "null", "uuid":
>>> > > > "1b0bf965-a391-478d-833d-3e37e4a15eb8", "networkofferingid": 1}
>>> with
>>> > > > requested profile NicProfile
>>> > > >
>>> > >
>>> >
>>> {"broadcastUri":null,"iPv4Address":null,"id":0,"reservationId":null,"vmId":0}
>>> > > > 2022-08-29 17:01:26,089 WARN [c.c.n.IpAddressManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to get ip
>>> address
>>> > > in
>>> > > > zone id=1, vlanId id=[3, 10, 13, 14]
>>> > > > 2022-08-29 17:01:26,090 DEBUG [c.c.u.d.T.Transaction]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Rolling back the
>>> > > > transaction: Time = 8 Name = consoleproxy-1; called by
>>> > > >
>>> > >
>>> >
>>> -TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-Transaction.execute:38-IpAddressManagerImpl.listAvailablePublicIps:824-IpAddressManagerImpl.fetchNewPublicIp:811-IpAddressManagerImpl.fetchNewPublicIp:803-IpAddressManagerImpl.assignPublicIpAddress:784-PublicNetworkGuru.getIp:125-PublicNetworkGuru.allocate:172-NetworkOrchestrator.allocateNic:973-NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice:823
>>> > > > 2022-08-29 17:01:26,090 WARN [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to allocate
>>> proxy
>>> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
>>> zone
>>> > [1]
>>> > > > due to [Insufficient address capacity].
>>> > > > com.cloud.exception.InsufficientAddressCapacityException:
>>> Insufficient
>>> > > > address capacityScope=interface com.cloud.dc.DataCenter; id=1
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
>>> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.IpAddressManagerImpl.listAvailablePublicIps(IpAddressManagerImpl.java:824)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl.java:811)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl.java:803)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.IpAddressManagerImpl.assignPublicIpAddress(IpAddressManagerImpl.java:784)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.guru.PublicNetworkGuru.getIp(PublicNetworkGuru.java:125)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.guru.PublicNetworkGuru.allocate(PublicNetworkGuru.java:172)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocateNic(NetworkOrchestrator.java:973)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice(NetworkOrchestrator.java:823)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$3.doInTransactionWithoutResult(NetworkOrchestrator.java:797)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
>>> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocate(NetworkOrchestrator.java:778)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl$1.doInTransactionWithoutResult(VirtualMachineManagerImpl.java:478)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
>>> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:469)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:528)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:700)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
>>> > > > at
>>> > >
>>> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
>>> > > > at
>>> > > >
>>> >
>>> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>>> > > > 2022-08-29 17:01:26,091 WARN [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to allocate
>>> > console
>>> > > > proxy standby capacity for zone [1] due to [Unable to allocate
>>> proxy
>>> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
>>> zone
>>> > [1]
>>> > > > due to [Insufficient address capacity].].
>>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to allocate
>>> > proxy
>>> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
>>> zone
>>> > [1]
>>> > > > due to [Insufficient address capacity].
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:704)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
>>> > > > at
>>> > >
>>> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
>>> > > > at
>>> > > >
>>> >
>>> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>>> > > > Caused by:
>>> com.cloud.exception.InsufficientAddressCapacityException:
>>> > > > Insufficient address capacityScope=interface
>>> com.cloud.dc.DataCenter;
>>> > > id=1
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
>>> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
>>> > > > ... 38 more
>>> > > > 2022-08-29 17:01:26,091 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) received console
>>> proxy
>>> > > alert
>>> > > > 2022-08-29 17:01:26,092 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Console proxy
>>> creation
>>> > > > failure, zone: xxxxxxxxxx
>>> > > > 2022-08-29 17:01:26,093 WARN [c.c.a.AlertManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) alertType=[10]
>>> > > > dataCenterId=[1] podId=[null] clusterId=[null] message=[Console
>>> proxy
>>> > > > creation failure. zone: xxxxxxxxxx, error details: Unable to
>>> allocate
>>> > > proxy
>>> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
>>> zone
>>> > [1]
>>> > > > due to [Insufficient address capacity].].
>>> > > > 2022-08-29 17:01:26,096 WARN [c.c.a.AlertManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) No recipients set in
>>> > > global
>>> > > > setting 'alert.email.addresses', skipping sending alert with
>>> subject
>>> > > > [Console proxy creation failure. zone: xxxxxxxxxx, error details:
>>> > Unable
>>> > > to
>>> > > > allocate proxy [Console VM instance {id: "85", name: "v-85-VM",
>>> uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
>>> zone
>>> > [1]
>>> > > > due to [Insufficient address capacity].] and content [Console proxy
>>> > > > creation failure (zone xxxxxxxxxx)].
>>> > > > 2022-08-29 17:01:26,096 WARN [c.c.v.SystemVmLoadScanner]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unexpected exception
>>> > > Unable
>>> > > > to allocate proxy [Console VM instance {id: "85", name: "v-85-VM",
>>> > uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
>>> zone
>>> > [1]
>>> > > > due to [Insufficient address capacity].
>>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to allocate
>>> > proxy
>>> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
>>> zone
>>> > [1]
>>> > > > due to [Insufficient address capacity].
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:704)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
>>> > > > at
>>> > >
>>> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
>>> > > > at
>>> > > >
>>> >
>>> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>>> > > > Caused by:
>>> com.cloud.exception.InsufficientAddressCapacityException:
>>> > > > Insufficient address capacityScope=interface
>>> com.cloud.dc.DataCenter;
>>> > > id=1
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
>>> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
>>> > > > ... 38 more
>>> > > > 2022-08-29 17:01:29,700 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-5:ctx-994465bd) (logid:1e2822fd) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:01:30,444 DEBUG
>>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
>>> > > > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 2
>>> routers
>>> > to
>>> > > > update status.
>>> > > > 2022-08-29 17:01:30,449 DEBUG
>>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
>>> > > > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 1
>>> VPC's to
>>> > > > update Redundant State.
>>> > > > 2022-08-29 17:01:30,451 DEBUG
>>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
>>> > > > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 0
>>> networks
>>> > to
>>> > > > update RvR status.
>>> > > > 2022-08-29 17:01:30,459 DEBUG [c.c.a.t.Request]
>>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
>>> > > > 6-4833769775052102463: Sending { Cmd , MgmtId: 90520740759984,
>>> via:
>>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
>>> > > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
>>> > > router.name
>>> > >
>>> >
>>> ":"r-79-VM","router.ip":"169.254.144.198"},"wait":"30","bypassHostMaintenance":"false"}}]
>>> > > > }
>>> > > > 2022-08-29 17:01:30,643 DEBUG [c.c.a.t.Request]
>>> > > > (AgentManager-Handler-12:null) (logid:) Seq 6-4833769775052102463:
>>> > > > Processing: { Ans: , MgmtId: 90520740759984, via: 6, Ver: v1,
>>> Flags:
>>> > 10,
>>> > > >
>>> > >
>>> >
>>> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"PRIMARY","result":"true","details":"Status:
>>> > > > PRIMARY
>>> > > > ","wait":"0","bypassHostMaintenance":"false"}}] }
>>> > > > 2022-08-29 17:01:30,643 DEBUG [c.c.a.t.Request]
>>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
>>> > > > 6-4833769775052102463: Received: { Ans: , MgmtId: 90520740759984,
>>> via:
>>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>>> CheckRouterAnswer } }
>>> > > > 2022-08-29 17:01:30,643 DEBUG [c.c.a.m.AgentManagerImpl]
>>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08)
>>> Details
>>> > > from
>>> > > > executing class com.cloud.agent.api.CheckRouterCommand: Status:
>>> PRIMARY
>>> > > >
>>> > > > 2022-08-29 17:01:30,655 DEBUG [c.c.a.t.Request]
>>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
>>> > > > 1-5410793477308947401: Sending { Cmd , MgmtId: 90520740759984,
>>> via:
>>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
>>> > > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
>>> > > router.name
>>> > >
>>> >
>>> ":"r-80-VM","router.ip":"169.254.217.200"},"wait":"30","bypassHostMaintenance":"false"}}]
>>> > > > }
>>> > > > 2022-08-29 17:01:30,796 DEBUG [c.c.a.t.Request]
>>> > > > (AgentManager-Handler-7:null) (logid:) Seq 1-5410793477308947401:
>>> > > > Processing: { Ans: , MgmtId: 90520740759984, via: 1, Ver: v1,
>>> Flags:
>>> > 10,
>>> > > >
>>> > >
>>> >
>>> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"BACKUP","result":"true","details":"Status:
>>> > > > BACKUP
>>> > > > ","wait":"0","bypassHostMaintenance":"false"}}] }
>>> > > > 2022-08-29 17:01:30,796 DEBUG [c.c.a.t.Request]
>>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
>>> > > > 1-5410793477308947401: Received: { Ans: , MgmtId: 90520740759984,
>>> via:
>>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>>> CheckRouterAnswer } }
>>> > > > 2022-08-29 17:01:30,796 DEBUG [c.c.a.m.AgentManagerImpl]
>>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08)
>>> Details
>>> > > from
>>> > > > executing class com.cloud.agent.api.CheckRouterCommand: Status:
>>> BACKUP
>>> > > >
>>> > > > 2022-08-29 17:01:33,703 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-5:ctx-70bca202) (logid:b332dbe0) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:01:34,083 DEBUG [c.c.s.StatsCollector]
>>> > > > (StatsCollector-6:ctx-213e2ecc) (logid:aa0584c9) AutoScaling
>>> Monitor is
>>> > > > running...
>>> > > > 2022-08-29 17:01:37,707 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-6:ctx-57493517) (logid:5093e75f) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:01:38,640 DEBUG [c.c.s.StatsCollector]
>>> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c)
>>> HostStatsCollector is
>>> > > > running...
>>> > > > 2022-08-29 17:01:38,655 DEBUG [c.c.a.t.Request]
>>> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
>>> > > 1-5410793477308947402:
>>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>>> GetHostStatsAnswer }
>>> > }
>>> > > > 2022-08-29 17:01:38,707 DEBUG [c.c.a.t.Request]
>>> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
>>> > > 5-8959067033723473395:
>>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>>> > > > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>>> GetHostStatsAnswer }
>>> > }
>>> > > > 2022-08-29 17:01:38,717 DEBUG [c.c.a.t.Request]
>>> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
>>> > > 6-4833769775052102464:
>>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>>> GetHostStatsAnswer }
>>> > }
>>> > > > 2022-08-29 17:01:38,725 DEBUG [c.c.a.t.Request]
>>> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
>>> > > 7-295548725546192472:
>>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>>> > > > 7(srv-5-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>>> GetHostStatsAnswer }
>>> > }
>>> > > > 2022-08-29 17:01:41,709 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-5:ctx-a65ae462) (logid:bc2b7320) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:01:45,712 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-2:ctx-05491ace) (logid:b2232c4d) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:01:47,019 DEBUG [c.c.a.m.AgentManagerImpl]
>>> > > > (AgentManager-Handler-14:null) (logid:) Ping from Routing host
>>> > > > 7(srv-5-comp.aaaaaaaa.bbb)
>>> > > >
>>> > > > 2022-08-29 17:01:49,715 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-2:ctx-7943581c) (logid:504c98ff) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:01:51,881 DEBUG [c.c.a.m.AgentManagerImpl]
>>> > > > (AgentManager-Handler-13:null) (logid:) Ping from Routing host
>>> > > > 1(srv-2-comp.aaaaaaaa.bbb)
>>> > > >
>>> > > > 2022-08-29 17:01:53,718 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-4:ctx-b11cfa22) (logid:3163613f) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:01:53,955 DEBUG [c.c.a.t.Request]
>>> > > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
>>> > > 1-5410793477308947403:
>>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer
>>> } }
>>> > > > 2022-08-29 17:01:54,025 DEBUG [c.c.a.t.Request]
>>> > > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
>>> > > 5-8959067033723473396:
>>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>>> > > > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer
>>> } }
>>> > > > 2022-08-29 17:01:54,096 DEBUG [c.c.a.t.Request]
>>> > > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
>>> > > 6-4833769775052102465:
>>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer
>>> } }
>>> > > > 2022-08-29 17:01:54,217 DEBUG [c.c.a.t.Request]
>>> > > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
>>> > > 7-295548725546192473:
>>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>>> > > > 7(srv-5-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer
>>> } }
>>> > > > 2022-08-29 17:01:55,765 DEBUG [c.c.s.StatsCollector]
>>> > > > (secstorage-1:ctx-7d9995d7) (logid:e388a8b8) Verifying image
>>> storage
>>> > [1].
>>> > > > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
>>> > > > 2022-08-29 17:01:55,771 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
>>> > > > (secstorage-1:ctx-7d9995d7) (logid:e388a8b8) Zone [1] is ready to
>>> > launch
>>> > > > secondary storage VM.
>>> > > > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Zone 1 is ready to
>>> > launch
>>> > > > console proxy
>>> > > > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Expand console proxy
>>> > > standby
>>> > > > capacity for zone xxxxxxxxxx
>>> > > > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Allocating console
>>> proxy
>>> > > > standby capacity for zone [1].
>>> > > > 2022-08-29 17:01:56,034 INFO [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Found a stopped
>>> console
>>> > > > proxy, starting it. Vm id : 85
>>> > > > 2022-08-29 17:01:56,035 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) start parameter
>>> value of
>>> > > > enterHardwareSetup == <very null> during processing of queued job
>>> > > > 2022-08-29 17:01:56,044 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Sync job-4288
>>> execution
>>> > on
>>> > > > object VmWorkJobQueue.85
>>> > > > 2022-08-29 17:01:56,173 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (AsyncJobMgr-Heartbeat-1:ctx-e75826b1) (logid:e80fe659) Execute
>>> > > sync-queue
>>> > > > item: SyncQueueItemVO {id:2105, queueId: 2105, contentType:
>>> AsyncJob,
>>> > > > contentId: 4288, lastProcessMsid: 90520740759984,
>>> lastprocessNumber: 1,
>>> > > > lastProcessTime: Mon Aug 29 17:01:56 CEST 2022, created: Mon Aug 29
>>> > > > 17:01:56 CEST 2022}
>>> > > > 2022-08-29 17:01:56,174 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (AsyncJobMgr-Heartbeat-1:ctx-e75826b1) (logid:e80fe659) Schedule
>>> queued
>>> > > > job-4288
>>> > > > 2022-08-29 17:01:56,179 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
>>> (logid:7e671dd0)
>>> > > Add
>>> > > > job-4288 into job monitoring
>>> > > > 2022-08-29 17:01:56,186 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
>>> (logid:ba10fec5)
>>> > > > Executing AsyncJobVO: {id:4288, userId: 1, accountId: 1,
>>> instanceType:
>>> > > > null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
>>> > > >
>>> > >
>>> >
>>> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAVXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
>>> > > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
>>> 0,
>>> > > > result: null, initMsid: 90520740759984, completeMsid: null,
>>> > lastUpdated:
>>> > > > null, lastPolled: null, created: Mon Aug 29 17:01:56 CEST 2022,
>>> > removed:
>>> > > > null}
>>> > > > 2022-08-29 17:01:56,186 DEBUG [c.c.v.VmWorkJobDispatcher]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
>>> (logid:ba10fec5)
>>> > > Run
>>> > > > VM work job: com.cloud.vm.VmWorkStart for VM 85, job origin: 3445
>>> > > > 2022-08-29 17:01:56,187 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Execute VM work job:
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
>>> > > > 2022-08-29 17:01:56,189 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) orchestrating VM start for 'v-85-VM'
>>> > > > com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
>>> > > > 2022-08-29 17:01:56,196 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
>>> > > > transited from [Stopped] to [Starting] with event [StartRequested].
>>> > VM's
>>> > > > original host: null, new host: null, host before state transition:
>>> null
>>> > > > 2022-08-29 17:01:56,196 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Successfully transitioned to start state for VM
>>> > instance
>>> > > > {id: "85", name: "v-85-VM", uuid:
>>> > "110d8620-c029-4ca9-9084-210099725a80",
>>> > > > type="ConsoleProxy"} reservation id =
>>> > > 0ab68202-d8be-469c-9b2a-87a55c2dc489
>>> > > > 2022-08-29 17:01:56,200 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts:
>>> null
>>> > > > 2022-08-29 17:01:56,202 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) VM start attempt #1
>>> > > > 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
>>> > > > 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Trying to allocate a host and storage pools from
>>> dc:1,
>>> > > > pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
>>> > > > 1073741824
>>> > > > 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
>>> > > > 2022-08-29 17:01:56,212 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
>>> > > > 2022-08-29 17:01:56,214 DEBUG [c.c.d.FirstFitPlanner]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Searching all possible resources under this Zone:
>>> 1
>>> > > > 2022-08-29 17:01:56,215 DEBUG [c.c.d.FirstFitPlanner]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Listing clusters in order of aggregate capacity,
>>> that
>>> > > have
>>> > > > (at least one host with) enough CPU and RAM capacity under this
>>> Zone: 1
>>> > > > 2022-08-29 17:01:56,218 DEBUG [c.c.d.FirstFitPlanner]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Removing from the clusterId list these clusters
>>> from
>>> > > avoid
>>> > > > set: []
>>> > > > 2022-08-29 17:01:56,224 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
>>> > > > 2022-08-29 17:01:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested
>>> with
>>> > > > Custom[UEFI] Boot Type false
>>> > > > 2022-08-29 17:01:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in
>>> dc: 1
>>> > > > pod:1 cluster:1
>>> > > > 2022-08-29 17:01:56,226 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4
>>> > hosts
>>> > > to
>>> > > > check for allocation: [Host {"id": "5", "name":
>>> > > "srv-4-comp.aaaaaaaa.bbb",
>>> > > > "uuid": "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"},
>>> Host
>>> > > > {"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
>>> {"id":
>>> > > "6",
>>> > > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
>>> {"id":
>>> > > "1",
>>> > > > "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
>>> > > > 2022-08-29 17:01:56,232 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for
>>> allocation
>>> > > > after prioritization: [Host {"id": "5", "name":
>>> > > "srv-4-comp.aaaaaaaa.bbb",
>>> > > > "uuid": "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"},
>>> Host
>>> > > > {"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
>>> {"id":
>>> > > "6",
>>> > > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
>>> {"id":
>>> > > "1",
>>> > > > "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
>>> > > > 2022-08-29 17:01:56,232 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for
>>> speed=500Mhz,
>>> > > > Ram=1024 MB
>>> > > > 2022-08-29 17:01:56,232 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
>>> > > > srv-4-comp.aaaaaaaa.bbb, uuid:
>>> 134c625d-15ab-461a-a38d-36fb0b979b5d} is
>>> > > KVM
>>> > > > hypervisor type, no max guest limit check needed
>>> > > > 2022-08-29 17:01:56,234 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu
>>> capability
>>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> speed:
>>> > 500
>>> > > > 2022-08-29 17:01:56,234 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has
>>> > enough
>>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > 1073741824 ,
>>> > > > cpuOverprovisioningFactor: 1.0
>>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> CPU:
>>> > > 127200
>>> > > > and CPU after applying overprovisioning: 127200
>>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 ,
>>> Requested
>>> > > > CPU: 500
>>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
>>> > > > 251009040384 , Requested RAM: (1.00 GB) 1073741824
>>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>>> RAM
>>> > > > available
>>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> from
>>> > > host:
>>> > > > 5, used: 11100, reserved: 0, actual total: 127200, total with
>>> > > > overprovisioning: 127200; requested
>>> cpu:500,alloc_from_last_host?:false
>>> > > > ,considerReservedCapacity?: true
>>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> from
>>> > > host:
>>> > > > 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total:
>>> (250.27
>>> > > GB)
>>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > alloc_from_last_host?:
>>> > > > false , considerReservedCapacity?: true
>>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>>> > adding
>>> > > to
>>> > > > list: 5
>>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
>>> > > > srv-5-comp.aaaaaaaa.bbb, uuid:
>>> 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is
>>> > > KVM
>>> > > > hypervisor type, no max guest limit check needed
>>> > > > 2022-08-29 17:01:56,238 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu
>>> capability
>>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> speed:
>>> > 500
>>> > > > 2022-08-29 17:01:56,238 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has
>>> > enough
>>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > 1073741824 ,
>>> > > > cpuOverprovisioningFactor: 1.0
>>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> CPU:
>>> > > 127200
>>> > > > and CPU after applying overprovisioning: 127200
>>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 ,
>>> Requested
>>> > > CPU:
>>> > > > 500
>>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
>>> > > > 189373743104 , Requested RAM: (1.00 GB) 1073741824
>>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>>> RAM
>>> > > > available
>>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> from
>>> > > host:
>>> > > > 7, used: 90100, reserved: 0, actual total: 127200, total with
>>> > > > overprovisioning: 127200; requested
>>> cpu:500,alloc_from_last_host?:false
>>> > > > ,considerReservedCapacity?: true
>>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> from
>>> > > host:
>>> > > > 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total:
>>> (250.27
>>> > > GB)
>>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > alloc_from_last_host?:
>>> > > > false , considerReservedCapacity?: true
>>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>>> > adding
>>> > > to
>>> > > > list: 7
>>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
>>> > > > srv-3-comp.aaaaaaaa.bbb, uuid:
>>> 6858713b-7e17-46d8-854d-24a13f9c7001} is
>>> > > KVM
>>> > > > hypervisor type, no max guest limit check needed
>>> > > > 2022-08-29 17:01:56,242 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu
>>> capability
>>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> speed:
>>> > 500
>>> > > > 2022-08-29 17:01:56,242 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has
>>> > enough
>>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > 1073741824 ,
>>> > > > cpuOverprovisioningFactor: 1.0
>>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> CPU:
>>> > > 127200
>>> > > > and CPU after applying overprovisioning: 127200
>>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 ,
>>> Requested
>>> > > CPU:
>>> > > > 500
>>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
>>> > > > 204032835584 , Requested RAM: (1.00 GB) 1073741824
>>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>>> RAM
>>> > > > available
>>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> from
>>> > > host:
>>> > > > 6, used: 42900, reserved: 0, actual total: 127200, total with
>>> > > > overprovisioning: 127200; requested
>>> cpu:500,alloc_from_last_host?:false
>>> > > > ,considerReservedCapacity?: true
>>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> from
>>> > > host:
>>> > > > 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total:
>>> (250.27
>>> > > GB)
>>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > alloc_from_last_host?:
>>> > > > false , considerReservedCapacity?: true
>>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>>> > adding
>>> > > to
>>> > > > list: 6
>>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
>>> > > > srv-2-comp.aaaaaaaa.bbb, uuid:
>>> 883f7f62-55f2-42f4-8242-16b6847c4204} is
>>> > > KVM
>>> > > > hypervisor type, no max guest limit check needed
>>> > > > 2022-08-29 17:01:56,246 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu
>>> capability
>>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> speed:
>>> > 500
>>> > > > 2022-08-29 17:01:56,246 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has
>>> > enough
>>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > 1073741824 ,
>>> > > > cpuOverprovisioningFactor: 1.0
>>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> CPU:
>>> > > 127200
>>> > > > and CPU after applying overprovisioning: 127200
>>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 ,
>>> Requested
>>> > > CPU:
>>> > > > 500
>>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
>>> > > > 229802639360 , Requested RAM: (1.00 GB) 1073741824
>>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>>> RAM
>>> > > > available
>>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> from
>>> > > host:
>>> > > > 1, used: 48200, reserved: 0, actual total: 127200, total with
>>> > > > overprovisioning: 127200; requested
>>> cpu:500,alloc_from_last_host?:false
>>> > > > ,considerReservedCapacity?: true
>>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> from
>>> > > host:
>>> > > > 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total:
>>> (250.27
>>> > > GB)
>>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > alloc_from_last_host?:
>>> > > > false , considerReservedCapacity?: true
>>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>>> > adding
>>> > > to
>>> > > > list: 1
>>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator
>>> returning 4
>>> > > > suitable hosts
>>> > > > 2022-08-29 17:01:56,258 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
>>> > > > transited from [Starting] to [Stopped] with event
>>> [OperationFailed].
>>> > VM's
>>> > > > original host: null, new host: null, host before state transition:
>>> null
>>> > > > 2022-08-29 17:01:56,258 WARN [c.c.v.VirtualMachineManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Unable to orchestrate start VM instance {id: "85",
>>> > name:
>>> > > > "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
>>> > > > type="ConsoleProxy"} due to [Unable to create deployment, no usable
>>> > > volumes
>>> > > > found for the VM: 85].
>>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to create
>>> > > > deployment, no usable volumes found for the VM: 85
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
>>> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>>> > Source)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>>> > > > at
>>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> > > > at
>>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>>> > > > 2022-08-29 17:01:56,259 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Done executing VM work job:
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
>>> > > > 2022-08-29 17:01:56,260 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Complete async job-4288, jobStatus: FAILED,
>>> > resultCode:
>>> > > 0,
>>> > > > result:
>>> > > >
>>> > >
>>> >
>>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODUiLCBuYW1lOiAidi04NS1WTSIsIHV1aWQ6ICIxMTBkODYyMC1jMDI5LTRjYTktOTA4NC0yMTAwOTk3MjVhODAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODVdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
>>> > > > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Publish async job-4288 complete on message bus
>>> > > > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Wake up jobs related to job-4288
>>> > > > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Update db status for job-4288
>>> > > > 2022-08-29 17:01:56,262 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Wake up jobs joined with job-4288 and disjoin all
>>> > > subjobs
>>> > > > created from job- 4288
>>> > > > 2022-08-29 17:01:56,270 DEBUG [c.c.v.VmWorkJobDispatcher]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
>>> (logid:ba10fec5)
>>> > > > Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 85,
>>> job
>>> > > > origin: 3445
>>> > > > 2022-08-29 17:01:56,270 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
>>> (logid:ba10fec5)
>>> > > > Done executing com.cloud.vm.VmWorkStart for job-4288
>>> > > > 2022-08-29 17:01:56,272 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
>>> (logid:ba10fec5)
>>> > > > Remove job-4288 from job monitoring
>>> > > > 2022-08-29 17:01:56,277 WARN [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Unable to allocate
>>> > console
>>> > > > proxy standby capacity for zone [1] due to [Unable to orchestrate
>>> start
>>> > > VM
>>> > > > instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>>> > > [Unable
>>> > > > to create deployment, no usable volumes found for the VM: 85].].
>>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to
>>> orchestrate
>>> > > > start VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>>> > > [Unable
>>> > > > to create deployment, no usable volumes found for the VM: 85].
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
>>> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>>> > Source)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>>> > > > at
>>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> > > > at
>>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>>> > > > 2022-08-29 17:01:56,277 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) received console
>>> proxy
>>> > > alert
>>> > > > 2022-08-29 17:01:56,278 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Console proxy
>>> creation
>>> > > > failure, zone: xxxxxxxxxx
>>> > > > 2022-08-29 17:01:56,279 WARN [c.c.a.AlertManagerImpl]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) alertType=[10]
>>> > > > dataCenterId=[1] podId=[null] clusterId=[null] message=[Console
>>> proxy
>>> > > > creation failure. zone: xxxxxxxxxx, error details: Unable to
>>> > orchestrate
>>> > > > start VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>>> > > [Unable
>>> > > > to create deployment, no usable volumes found for the VM: 85].].
>>> > > > 2022-08-29 17:01:56,284 WARN [c.c.a.AlertManagerImpl]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) No recipients set in
>>> > > global
>>> > > > setting 'alert.email.addresses', skipping sending alert with
>>> subject
>>> > > > [Console proxy creation failure. zone: xxxxxxxxxx, error details:
>>> > Unable
>>> > > to
>>> > > > orchestrate start VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>>> > > [Unable
>>> > > > to create deployment, no usable volumes found for the VM: 85].] and
>>> > > content
>>> > > > [Console proxy creation failure (zone xxxxxxxxxx)].
>>> > > > 2022-08-29 17:01:56,284 WARN [c.c.v.SystemVmLoadScanner]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Unexpected exception
>>> > > Unable
>>> > > > to orchestrate start VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>>> > > [Unable
>>> > > > to create deployment, no usable volumes found for the VM: 85].
>>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to
>>> orchestrate
>>> > > > start VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>>> > > [Unable
>>> > > > to create deployment, no usable volumes found for the VM: 85].
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
>>> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>>> > Source)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>>> > > > at
>>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> > > > at
>>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>>> > > > 2022-08-29 17:01:57,721 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-4:ctx-9528226f) (logid:4512ff9b) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:01:59,504 DEBUG [c.c.s.StatsCollector]
>>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) StorageCollector
>>> is
>>> > > > running...
>>> > > > 2022-08-29 17:01:59,518 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
>>> > > getCommandHostDelegation:
>>> > > > class com.cloud.agent.api.GetStorageStatsCommand
>>> > > > 2022-08-29 17:01:59,518 DEBUG [c.c.h.XenServerGuru]
>>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning
>>> the
>>> > > > default host to execute commands because the command is not of Copy
>>> > type.
>>> > > > 2022-08-29 17:01:59,571 DEBUG [c.c.a.t.Request]
>>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
>>> > > > 11-7523826127475836583: Received: { Ans: , MgmtId: 90520740759984,
>>> > via:
>>> > > > 11(s-32-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
>>> > > > 2022-08-29 17:01:59,574 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
>>> > > getCommandHostDelegation:
>>> > > > class com.cloud.agent.api.GetStorageStatsCommand
>>> > > > 2022-08-29 17:01:59,574 DEBUG [c.c.h.XenServerGuru]
>>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning
>>> the
>>> > > > default host to execute commands because the command is not of Copy
>>> > type.
>>> > > > 2022-08-29 17:01:59,615 DEBUG [c.c.a.t.Request]
>>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
>>> > > 5-8959067033723473397:
>>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>>> > > > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>>> GetStorageStatsAnswer
>>> > > } }
>>> > > > 2022-08-29 17:01:59,624 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
>>> > > getCommandHostDelegation:
>>> > > > class com.cloud.agent.api.GetStorageStatsCommand
>>> > > > 2022-08-29 17:01:59,624 DEBUG [c.c.h.XenServerGuru]
>>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning
>>> the
>>> > > > default host to execute commands because the command is not of Copy
>>> > type.
>>> > > > 2022-08-29 17:01:59,653 DEBUG [c.c.a.t.Request]
>>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
>>> > > 6-4833769775052102466:
>>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>>> GetStorageStatsAnswer
>>> > > } }
>>> > > > 2022-08-29 17:01:59,979 DEBUG [c.c.a.m.AgentManagerImpl]
>>> > > > (AgentManager-Handler-12:null) (logid:) Ping from Routing host
>>> > > > 5(srv-4-comp.aaaaaaaa.bbb)
>>> > > >
>>> > > > 2022-08-29 17:02:00,269 DEBUG [c.c.s.StatsCollector]
>>> > > > (StatsCollector-5:ctx-d2b6ffd3) (logid:95201678) DbCollector is
>>> > > running...
>>> > > > 2022-08-29 17:02:00,443 DEBUG
>>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
>>> > > > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 2
>>> routers
>>> > to
>>> > > > update status.
>>> > > > 2022-08-29 17:02:00,449 DEBUG
>>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
>>> > > > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 1
>>> VPC's to
>>> > > > update Redundant State.
>>> > > > 2022-08-29 17:02:00,450 DEBUG
>>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
>>> > > > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 0
>>> networks
>>> > to
>>> > > > update RvR status.
>>> > > > 2022-08-29 17:02:00,457 DEBUG [c.c.a.t.Request]
>>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
>>> > > > 6-4833769775052102467: Sending { Cmd , MgmtId: 90520740759984,
>>> via:
>>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
>>> > > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
>>> > > router.name
>>> > >
>>> >
>>> ":"r-79-VM","router.ip":"169.254.144.198"},"wait":"30","bypassHostMaintenance":"false"}}]
>>> > > > }
>>> > > > 2022-08-29 17:02:00,599 DEBUG [c.c.a.t.Request]
>>> > > > (AgentManager-Handler-7:null) (logid:) Seq 6-4833769775052102467:
>>> > > > Processing: { Ans: , MgmtId: 90520740759984, via: 6, Ver: v1,
>>> Flags:
>>> > 10,
>>> > > >
>>> > >
>>> >
>>> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"PRIMARY","result":"true","details":"Status:
>>> > > > PRIMARY
>>> > > > ","wait":"0","bypassHostMaintenance":"false"}}] }
>>> > > > 2022-08-29 17:02:00,599 DEBUG [c.c.a.t.Request]
>>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
>>> > > > 6-4833769775052102467: Received: { Ans: , MgmtId: 90520740759984,
>>> via:
>>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>>> CheckRouterAnswer } }
>>> > > > 2022-08-29 17:02:00,599 DEBUG [c.c.a.m.AgentManagerImpl]
>>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f)
>>> Details
>>> > > > from executing class com.cloud.agent.api.CheckRouterCommand:
>>> Status:
>>> > > PRIMARY
>>> > > >
>>> > > > 2022-08-29 17:02:00,612 DEBUG [c.c.a.t.Request]
>>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
>>> > > > 1-5410793477308947404: Sending { Cmd , MgmtId: 90520740759984,
>>> via:
>>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
>>> > > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
>>> > > router.name
>>> > >
>>> >
>>> ":"r-80-VM","router.ip":"169.254.217.200"},"wait":"30","bypassHostMaintenance":"false"}}]
>>> > > > }
>>> > > > 2022-08-29 17:02:00,748 DEBUG [c.c.a.m.AgentManagerImpl]
>>> > > > (AgentManager-Handler-4:null) (logid:) Ping from 11(s-32-VM)
>>> > > > 2022-08-29 17:02:00,750 DEBUG [c.c.a.t.Request]
>>> > > > (AgentManager-Handler-1:null) (logid:) Seq 1-5410793477308947404:
>>> > > > Processing: { Ans: , MgmtId: 90520740759984, via: 1, Ver: v1,
>>> Flags:
>>> > 10,
>>> > > >
>>> > >
>>> >
>>> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"BACKUP","result":"true","details":"Status:
>>> > > > BACKUP
>>> > > > ","wait":"0","bypassHostMaintenance":"false"}}] }
>>> > > > 2022-08-29 17:02:00,750 DEBUG [c.c.a.t.Request]
>>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
>>> > > > 1-5410793477308947404: Received: { Ans: , MgmtId: 90520740759984,
>>> via:
>>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>>> CheckRouterAnswer } }
>>> > > > 2022-08-29 17:02:00,751 DEBUG [c.c.a.m.AgentManagerImpl]
>>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f)
>>> Details
>>> > > > from executing class com.cloud.agent.api.CheckRouterCommand:
>>> Status:
>>> > > BACKUP
>>> > > >
>>> > > > 2022-08-29 17:02:01,362 DEBUG [c.c.s.StatsCollector]
>>> > > > (Cluster-Worker-839:ctx-e097a8ab) (logid:3deacf65) StatusUpdate
>>> from
>>> > > > 90520740759984, json:
>>> > > >
>>> > >
>>> >
>>> {"managementServerHostId":1,"managementServerHostUuid":"5615ca5f-0716-4da1-93ec-31f79fd19bbe","collectionTime":"Aug
>>> > > > 29, 2022, 5:02:01
>>> > > >
>>> > >
>>> >
>>> PM","sessions":1,"cpuUtilization":0.0,"totalJvmMemoryBytes":324534272,"freeJvmMemoryBytes":68595424,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":103760388,"jvmStartTime":1661681560910,"availableProcessors":6,"loadAverage":0.02,"totalInit":139788288,"totalUsed":474791104,"totalCommitted":557662208,"pid":1522332,"jvmName":"1522332@vm-1-cs-mgmt.aaaaaaaa.bbb
>>> > >
>>> >
>>> ","jvmVendor":"Ubuntu","jvmVersion":"11.0.16+8-post-Ubuntu-0ubuntu120.04","osDistribution":"Ubuntu
>>> > > > 20.04.5
>>> > > >
>>> > >
>>> >
>>> LTS","agentCount":5,"heapMemoryUsed":256775312,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":13,"threadsRunnableCount":15,"threadsTerminatedCount":0,"threadsTotalCount":420,"threadsWaitingCount":331,"systemMemoryTotal":8343236608,"systemMemoryFree":2623193088,"systemMemoryUsed":814808,"systemMemoryVirtualSize":7494184960,"logInfo":"/var/log/cloudstack/management/management-server.log
>>> > > > using: 74M\non disk /dev/vda2 mounted on / (14%
>>> > > >
>>> > >
>>> >
>>> full)","systemTotalCpuCycles":15870.180000000002,"systemLoadAverages":[0.02,0.03,0.0],"systemCyclesUsage":[4116569,2615101,2716460424],"dbLocal":false,"usageLocal":false,"systemBootTime":"Jul
>>> > > > 8, 2022, 2:35:55 AM","kernelVersion":"5.4.0-121-generic"}
>>> > > > 2022-08-29 17:02:01,723 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-4:ctx-7abb1b5d) (logid:2320950d) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:02:05,417 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (Vm-Operations-Cleanup-1:ctx-e741e208) (logid:3f5129e0) VM
>>> Operation
>>> > > Thread
>>> > > > Running
>>> > > > 2022-08-29 17:02:05,725 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-4:ctx-8e7f6766) (logid:710cb3b2) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:02:08,626 DEBUG [c.c.a.m.AgentManagerImpl]
>>> > > > (AgentManager-Handler-11:null) (logid:) Ping from Routing host
>>> > > > 6(srv-3-comp.aaaaaaaa.bbb)
>>> > > >
>>> > > > 2022-08-29 17:02:09,727 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-4:ctx-26a5ff07) (logid:09b3c89a) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:02:13,729 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-4:ctx-b4d4e374) (logid:9048c7a7) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:02:17,731 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-4:ctx-3db3d208) (logid:7bb73f7b) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:02:21,734 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-1:ctx-fca1d248) (logid:1ed5381a) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:02:25,737 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-1:ctx-984e2382) (logid:e6c3d724) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:02:25,759 DEBUG [c.c.s.StatsCollector]
>>> > > > (secstorage-1:ctx-64c610ed) (logid:6603cd38) Verifying image
>>> storage
>>> > [1].
>>> > > > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
>>> > > > 2022-08-29 17:02:25,763 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
>>> > > > (secstorage-1:ctx-64c610ed) (logid:6603cd38) Zone [1] is ready to
>>> > launch
>>> > > > secondary storage VM.
>>> > > > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Zone 1 is ready to
>>> > launch
>>> > > > console proxy
>>> > > > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Expand console proxy
>>> > > standby
>>> > > > capacity for zone xxxxxxxxxx
>>> > > > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Allocating console
>>> proxy
>>> > > > standby capacity for zone [1].
>>> > > > 2022-08-29 17:02:26,030 INFO [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Found a stopped
>>> console
>>> > > > proxy, starting it. Vm id : 85
>>> > > > 2022-08-29 17:02:26,031 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) start parameter
>>> value of
>>> > > > enterHardwareSetup == <very null> during processing of queued job
>>> > > > 2022-08-29 17:02:26,037 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Sync job-4289
>>> execution
>>> > on
>>> > > > object VmWorkJobQueue.85
>>> > > > 2022-08-29 17:02:26,167 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (AsyncJobMgr-Heartbeat-1:ctx-bba6b89d) (logid:f9aed0eb) Execute
>>> > > sync-queue
>>> > > > item: SyncQueueItemVO {id:2106, queueId: 2105, contentType:
>>> AsyncJob,
>>> > > > contentId: 4289, lastProcessMsid: 90520740759984,
>>> lastprocessNumber: 2,
>>> > > > lastProcessTime: Mon Aug 29 17:02:26 CEST 2022, created: Mon Aug 29
>>> > > > 17:02:26 CEST 2022}
>>> > > > 2022-08-29 17:02:26,167 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (AsyncJobMgr-Heartbeat-1:ctx-bba6b89d) (logid:f9aed0eb) Schedule
>>> queued
>>> > > > job-4289
>>> > > > 2022-08-29 17:02:26,172 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
>>> (logid:f919f9b3)
>>> > > Add
>>> > > > job-4289 into job monitoring
>>> > > > 2022-08-29 17:02:26,181 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
>>> (logid:ba10fec5)
>>> > > > Executing AsyncJobVO: {id:4289, userId: 1, accountId: 1,
>>> instanceType:
>>> > > > null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
>>> > > >
>>> > >
>>> >
>>> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAVXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
>>> > > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
>>> 0,
>>> > > > result: null, initMsid: 90520740759984, completeMsid: null,
>>> > lastUpdated:
>>> > > > null, lastPolled: null, created: Mon Aug 29 17:02:26 CEST 2022,
>>> > removed:
>>> > > > null}
>>> > > > 2022-08-29 17:02:26,181 DEBUG [c.c.v.VmWorkJobDispatcher]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
>>> (logid:ba10fec5)
>>> > > Run
>>> > > > VM work job: com.cloud.vm.VmWorkStart for VM 85, job origin: 3445
>>> > > > 2022-08-29 17:02:26,183 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Execute VM work job:
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
>>> > > > 2022-08-29 17:02:26,184 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) orchestrating VM start for 'v-85-VM'
>>> > > > com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
>>> > > > 2022-08-29 17:02:26,192 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
>>> > > > transited from [Stopped] to [Starting] with event [StartRequested].
>>> > VM's
>>> > > > original host: null, new host: null, host before state transition:
>>> null
>>> > > > 2022-08-29 17:02:26,192 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Successfully transitioned to start state for VM
>>> > instance
>>> > > > {id: "85", name: "v-85-VM", uuid:
>>> > "110d8620-c029-4ca9-9084-210099725a80",
>>> > > > type="ConsoleProxy"} reservation id =
>>> > > 24bf955f-96d8-4f4f-ae33-2cae2f3e0425
>>> > > > 2022-08-29 17:02:26,198 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts:
>>> null
>>> > > > 2022-08-29 17:02:26,200 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) VM start attempt #1
>>> > > > 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
>>> > > > 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Trying to allocate a host and storage pools from
>>> dc:1,
>>> > > > pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
>>> > > > 1073741824
>>> > > > 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
>>> > > > 2022-08-29 17:02:26,211 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
>>> > > > 2022-08-29 17:02:26,212 DEBUG [c.c.d.FirstFitPlanner]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Searching all possible resources under this Zone:
>>> 1
>>> > > > 2022-08-29 17:02:26,213 DEBUG [c.c.d.FirstFitPlanner]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Listing clusters in order of aggregate capacity,
>>> that
>>> > > have
>>> > > > (at least one host with) enough CPU and RAM capacity under this
>>> Zone: 1
>>> > > > 2022-08-29 17:02:26,216 DEBUG [c.c.d.FirstFitPlanner]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Removing from the clusterId list these clusters
>>> from
>>> > > avoid
>>> > > > set: []
>>> > > > 2022-08-29 17:02:26,221 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
>>> > > > 2022-08-29 17:02:26,222 INFO [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested
>>> with
>>> > > > Custom[UEFI] Boot Type false
>>> > > > 2022-08-29 17:02:26,223 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in
>>> dc: 1
>>> > > > pod:1 cluster:1
>>> > > > 2022-08-29 17:02:26,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4
>>> > hosts
>>> > > to
>>> > > > check for allocation: [Host {"id": "7", "name":
>>> > > "srv-5-comp.aaaaaaaa.bbb",
>>> > > > "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"},
>>> Host
>>> > > > {"id": "1", "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}, Host
>>> {"id":
>>> > > "6",
>>> > > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
>>> {"id":
>>> > > "5",
>>> > > > "name": "srv-4-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}]
>>> > > > 2022-08-29 17:02:26,231 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for
>>> allocation
>>> > > > after prioritization: [Host {"id": "7", "name":
>>> > > "srv-5-comp.aaaaaaaa.bbb",
>>> > > > "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"},
>>> Host
>>> > > > {"id": "1", "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}, Host
>>> {"id":
>>> > > "6",
>>> > > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
>>> {"id":
>>> > > "5",
>>> > > > "name": "srv-4-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}]
>>> > > > 2022-08-29 17:02:26,231 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for
>>> speed=500Mhz,
>>> > > > Ram=1024 MB
>>> > > > 2022-08-29 17:02:26,231 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
>>> > > > srv-5-comp.aaaaaaaa.bbb, uuid:
>>> 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is
>>> > > KVM
>>> > > > hypervisor type, no max guest limit check needed
>>> > > > 2022-08-29 17:02:26,234 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu
>>> capability
>>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> speed:
>>> > 500
>>> > > > 2022-08-29 17:02:26,234 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has
>>> > enough
>>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > 1073741824 ,
>>> > > > cpuOverprovisioningFactor: 1.0
>>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> CPU:
>>> > > 127200
>>> > > > and CPU after applying overprovisioning: 127200
>>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 ,
>>> Requested
>>> > > CPU:
>>> > > > 500
>>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
>>> > > > 189373743104 , Requested RAM: (1.00 GB) 1073741824
>>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>>> RAM
>>> > > > available
>>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> from
>>> > > host:
>>> > > > 7, used: 90100, reserved: 0, actual total: 127200, total with
>>> > > > overprovisioning: 127200; requested
>>> cpu:500,alloc_from_last_host?:false
>>> > > > ,considerReservedCapacity?: true
>>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> from
>>> > > host:
>>> > > > 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total:
>>> (250.27
>>> > > GB)
>>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > alloc_from_last_host?:
>>> > > > false , considerReservedCapacity?: true
>>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>>> > adding
>>> > > to
>>> > > > list: 7
>>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
>>> > > > srv-2-comp.aaaaaaaa.bbb, uuid:
>>> 883f7f62-55f2-42f4-8242-16b6847c4204} is
>>> > > KVM
>>> > > > hypervisor type, no max guest limit check needed
>>> > > > 2022-08-29 17:02:26,239 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu
>>> capability
>>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> speed:
>>> > 500
>>> > > > 2022-08-29 17:02:26,239 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has
>>> > enough
>>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > 1073741824 ,
>>> > > > cpuOverprovisioningFactor: 1.0
>>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> CPU:
>>> > > 127200
>>> > > > and CPU after applying overprovisioning: 127200
>>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 ,
>>> Requested
>>> > > CPU:
>>> > > > 500
>>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
>>> > > > 229802639360 , Requested RAM: (1.00 GB) 1073741824
>>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>>> RAM
>>> > > > available
>>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> from
>>> > > host:
>>> > > > 1, used: 48200, reserved: 0, actual total: 127200, total with
>>> > > > overprovisioning: 127200; requested
>>> cpu:500,alloc_from_last_host?:false
>>> > > > ,considerReservedCapacity?: true
>>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> from
>>> > > host:
>>> > > > 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total:
>>> (250.27
>>> > > GB)
>>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > alloc_from_last_host?:
>>> > > > false , considerReservedCapacity?: true
>>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>>> > adding
>>> > > to
>>> > > > list: 1
>>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
>>> > > > srv-3-comp.aaaaaaaa.bbb, uuid:
>>> 6858713b-7e17-46d8-854d-24a13f9c7001} is
>>> > > KVM
>>> > > > hypervisor type, no max guest limit check needed
>>> > > > 2022-08-29 17:02:26,243 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu
>>> capability
>>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> speed:
>>> > 500
>>> > > > 2022-08-29 17:02:26,243 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has
>>> > enough
>>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > 1073741824 ,
>>> > > > cpuOverprovisioningFactor: 1.0
>>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> CPU:
>>> > > 127200
>>> > > > and CPU after applying overprovisioning: 127200
>>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 ,
>>> Requested
>>> > > CPU:
>>> > > > 500
>>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
>>> > > > 204032835584 , Requested RAM: (1.00 GB) 1073741824
>>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>>> RAM
>>> > > > available
>>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> from
>>> > > host:
>>> > > > 6, used: 42900, reserved: 0, actual total: 127200, total with
>>> > > > overprovisioning: 127200; requested
>>> cpu:500,alloc_from_last_host?:false
>>> > > > ,considerReservedCapacity?: true
>>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> from
>>> > > host:
>>> > > > 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total:
>>> (250.27
>>> > > GB)
>>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > alloc_from_last_host?:
>>> > > > false , considerReservedCapacity?: true
>>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>>> > adding
>>> > > to
>>> > > > list: 6
>>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
>>> > > > srv-4-comp.aaaaaaaa.bbb, uuid:
>>> 134c625d-15ab-461a-a38d-36fb0b979b5d} is
>>> > > KVM
>>> > > > hypervisor type, no max guest limit check needed
>>> > > > 2022-08-29 17:02:26,247 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu
>>> capability
>>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> speed:
>>> > 500
>>> > > > 2022-08-29 17:02:26,247 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has
>>> > enough
>>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > 1073741824 ,
>>> > > > cpuOverprovisioningFactor: 1.0
>>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> CPU:
>>> > > 127200
>>> > > > and CPU after applying overprovisioning: 127200
>>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 ,
>>> Requested
>>> > > > CPU: 500
>>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
>>> > > > 251009040384 , Requested RAM: (1.00 GB) 1073741824
>>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>>> RAM
>>> > > > available
>>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> from
>>> > > host:
>>> > > > 5, used: 11100, reserved: 0, actual total: 127200, total with
>>> > > > overprovisioning: 127200; requested
>>> cpu:500,alloc_from_last_host?:false
>>> > > > ,considerReservedCapacity?: true
>>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> from
>>> > > host:
>>> > > > 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total:
>>> (250.27
>>> > > GB)
>>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > alloc_from_last_host?:
>>> > > > false , considerReservedCapacity?: true
>>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>>> > adding
>>> > > to
>>> > > > list: 5
>>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator
>>> returning 4
>>> > > > suitable hosts
>>> > > > 2022-08-29 17:02:26,260 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
>>> > > > transited from [Starting] to [Stopped] with event
>>> [OperationFailed].
>>> > VM's
>>> > > > original host: null, new host: null, host before state transition:
>>> null
>>> > > > 2022-08-29 17:02:26,260 WARN [c.c.v.VirtualMachineManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Unable to orchestrate start VM instance {id: "85",
>>> > name:
>>> > > > "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
>>> > > > type="ConsoleProxy"} due to [Unable to create deployment, no usable
>>> > > volumes
>>> > > > found for the VM: 85].
>>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to create
>>> > > > deployment, no usable volumes found for the VM: 85
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
>>> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>>> > Source)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>>> > > > at
>>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> > > > at
>>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>>> > > > 2022-08-29 17:02:26,261 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Done executing VM work job:
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
>>> > > > 2022-08-29 17:02:26,262 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Complete async job-4289, jobStatus: FAILED,
>>> > resultCode:
>>> > > 0,
>>> > > > result:
>>> > > >
>>> > >
>>> >
>>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODUiLCBuYW1lOiAidi04NS1WTSIsIHV1aWQ6ICIxMTBkODYyMC1jMDI5LTRjYTktOTA4NC0yMTAwOTk3MjVhODAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODVdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
>>> > > > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Publish async job-4289 complete on message bus
>>> > > > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Wake up jobs related to job-4289
>>> > > > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Update db status for job-4289
>>> > > > 2022-08-29 17:02:26,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Wake up jobs joined with job-4289 and disjoin all
>>> > > subjobs
>>> > > > created from job- 4289
>>> > > > 2022-08-29 17:02:26,269 DEBUG [c.c.v.VmWorkJobDispatcher]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
>>> (logid:ba10fec5)
>>> > > > Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 85,
>>> job
>>> > > > origin: 3445
>>> > > > 2022-08-29 17:02:26,270 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
>>> (logid:ba10fec5)
>>> > > > Done executing com.cloud.vm.VmWorkStart for job-4289
>>> > > > 2022-08-29 17:02:26,271 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
>>> (logid:ba10fec5)
>>> > > > Remove job-4289 from job monitoring
>>> > > > 2022-08-29 17:02:26,277 WARN [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Unable to allocate
>>> > console
>>> > > > proxy standby capacity for zone [1] due to [Unable to orchestrate
>>> start
>>> > > VM
>>> > > > instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>>> > > [Unable
>>> > > > to create deployment, no usable volumes found for the VM: 85].].
>>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to
>>> orchestrate
>>> > > > start VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>>> > > [Unable
>>> > > > to create deployment, no usable volumes found for the VM: 85].
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
>>> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>>> > Source)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>>> > > > at
>>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> > > > at
>>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>>> > > > 2022-08-29 17:02:26,277 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) received console
>>> proxy
>>> > > alert
>>> > > >
>>> > > >
>>> > > >
>>> > > > Am Mo., 29. Aug. 2022 um 16:30 Uhr schrieb Slavka Peleva <
>>> > > > slavkap@storpool.com>:
>>> > > >
>>> > > >> Thanks, Chris, for the information!
>>> > > >>
>>> > > >> Are you using custom offerings for the system VMs? Can you share
>>> more
>>> > > >> info if yes?
>>> > > >>
>>> > > >> About the log, it's not the complete log from where the deployment
>>> > > >> started. The deployment begins a bit earlier than what you
>>> shared. Can
>>> > > you
>>> > > >> search the management log for `Sync job-3539 execution on object
>>> > > >> VmWorkJobQueue` to get the `logid` for the volume creation
>>> (coloured
>>> > in
>>> > > >> blue)? And if possible, share the logs with the found log ID
>>> > > >>
>>> > > >> Here is an example from my logs:
>>> > > >>
>>> > > >> 2022-08-29 17:07:49,114 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > >> (consoleproxy-1:ctx-a952a3dc) (logid:89157bb9) Sync job-60484
>>> > execution
>>> > > >> on object VmWorkJobQueue.4904
>>> > > >>
>>> > > >> 2022-08-29 17:07:50,816 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>>> > > >> (Work-Job-Executor-5:ctx-56172581 job-60479/job-60484
>>> ctx-a900986d)
>>> > > >> (logid:95d750ae) Execute VM work job:
>>> > > >>
>>> > >
>>> >
>>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":4904,"handlerName":"VirtualMachineManagerImpl"}
>>> > > >> 2022-08-29 17:07:50,817 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > >> (Work-Job-Executor-5:ctx-56172581 job-60479/job-60484
>>> ctx-a900986d)
>>> > > >> (logid:95d750ae) orchestrating VM start for 'v-4904-VM'
>>> > > >> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
>>> > > >>
>>> > > >>
>>> > > >> On Mon, Aug 29, 2022 at 2:29 PM Vash_X@gmx.de <Va...@gmx.de>
>>> wrote:
>>> > > >>
>>> > > >>> Hi Slavka,
>>> > > >>>
>>> > > >>> thats what i did after your suggestion / informatin and where the
>>> > error
>>> > > >>> occured.
>>> > > >>> Deleted / Destroyed the consoleproxy and waited till recreation.
>>> > Which
>>> > > >>> didn't worked throwing the error i pasted above.
>>> > > >>> Sorry for beeing so unspecific.
>>> > > >>>
>>> > > >>> Regards,
>>> > > >>> Chris
>>> > > >>>
>>> > > >>> Am Mo., 29. Aug. 2022 um 12:46 Uhr schrieb Slavka Peleva <
>>> > > >>> slavkap@storpool.com>:
>>> > > >>>
>>> > > >>>> Hi Chris,
>>> > > >>>>
>>> > > >>>> I mean to recreate the system VMs by destroying them.
>>> > > >>>>
>>> > > >>>> On Mon, Aug 29, 2022 at 12:38 PM Vash_X@gmx.de <Va...@gmx.de>
>>> > wrote:
>>> > > >>>>
>>> > > >>>>> Hi Slavka,
>>> > > >>>>>
>>> > > >>>>> didn't tried to restart the SystemVMs. Reading through the
>>> docs i
>>> > had
>>> > > >>>>> the impression that "live-patching" would be enough for
>>> this....
>>> > > >>>>> Anyway - I tried my luck with the console proxy.
>>> > > >>>>>
>>> > > >>>>> Which failed utterly. Not starting as it isn't finding any
>>> useable
>>> > > >>>>> storage pool. - logs at the end of the mail.
>>> > > >>>>>
>>> > > >>>>> I tried deploying a VM using the same storage pool which worked
>>> > > >>>>> wonderfully.
>>> > > >>>>> There i noticed that all my custome storage-offerings are
>>> unuseable
>>> > > at
>>> > > >>>>> the moment. They are not accepting any custome storage sizes
>>> > > anymore...
>>> > > >>>>> Throwing failures "VM Creation failed. Volume size: -1GB is
>>> out of
>>> > > allowed
>>> > > >>>>> tange. Max:3072 Min:1"...
>>> > > >>>>> So this is also currently broken....
>>> > > >>>>>
>>> > > >>>>> 2022-08-29 10:51:56,189 DEBUG [c.c.v.VmWorkJobDispatcher]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
>>> > > (logid:ba10fec5) Run
>>> > > >>>>> VM work job: com.cloud.vm.VmWorkStart for VM 83, job origin:
>>> 3445
>>> > > >>>>> 2022-08-29 10:51:56,191 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Execute VM work job:
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
>>> > > >>>>> 2022-08-29 10:51:56,192 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) orchestrating VM start for 'v-83-VM'
>>> > > >>>>> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
>>> > > >>>>> 2022-08-29 10:51:56,198 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
>>> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
>>> state
>>> > > >>>>> transited from [Stopped] to [Starting] with event
>>> [StartRequested].
>>> > > VM's
>>> > > >>>>> original host: null, new host: null, host before state
>>> transition:
>>> > > null
>>> > > >>>>> 2022-08-29 10:51:56,198 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Successfully transitioned to start state for
>>> VM
>>> > > instance
>>> > > >>>>> {id: "83", name: "v-83-VM", uuid:
>>> > > "b1d1f9c6-eafe-4527-9335-e649c646aab0",
>>> > > >>>>> type="ConsoleProxy"} reservation id =
>>> > > 6641ccc0-bcef-4b1a-a1f7-8dd8000626ad
>>> > > >>>>> 2022-08-29 10:51:56,203 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Deploy avoids pods: null, clusters: null,
>>> hosts:
>>> > > null
>>> > > >>>>> 2022-08-29 10:51:56,206 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) VM start attempt #1
>>> > > >>>>> 2022-08-29 10:51:56,207 DEBUG
>>> [c.c.d.DeploymentPlanningManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
>>> > > >>>>> 2022-08-29 10:51:56,207 DEBUG
>>> [c.c.d.DeploymentPlanningManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Trying to allocate a host and storage pools
>>> from
>>> > > dc:1,
>>> > > >>>>> pod:null,cluster:null, requested cpu: 500, requested ram:
>>> (1.00 GB)
>>> > > >>>>> 1073741824
>>> > > >>>>> 2022-08-29 10:51:56,207 DEBUG
>>> [c.c.d.DeploymentPlanningManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Is ROOT volume READY (pool already
>>> allocated)?: No
>>> > > >>>>> 2022-08-29 10:51:56,214 DEBUG
>>> [c.c.d.DeploymentPlanningManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts:
>>> null
>>> > > >>>>> 2022-08-29 10:51:56,215 DEBUG [c.c.d.FirstFitPlanner]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Searching all possible resources under this
>>> Zone:
>>> > 1
>>> > > >>>>> 2022-08-29 10:51:56,216 DEBUG [c.c.d.FirstFitPlanner]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Listing clusters in order of aggregate
>>> capacity,
>>> > > that have
>>> > > >>>>> (at least one host with) enough CPU and RAM capacity under this
>>> > > Zone: 1
>>> > > >>>>> 2022-08-29 10:51:56,219 DEBUG [c.c.d.FirstFitPlanner]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Removing from the clusterId list these
>>> clusters
>>> > > from avoid
>>> > > >>>>> set: []
>>> > > >>>>> 2022-08-29 10:51:56,224 DEBUG
>>> [c.c.d.DeploymentPlanningManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
>>> > > >>>>> 2022-08-29 10:51:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is
>>> requested
>>> > > with
>>> > > >>>>> Custom[UEFI] Boot Type false
>>> > > >>>>> 2022-08-29 10:51:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in
>>> > dc: 1
>>> > > >>>>> pod:1 cluster:1
>>> > > >>>>> 2022-08-29 10:51:56,227 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator
>>> has 4
>>> > > hosts to
>>> > > >>>>> check for allocation: [Host {"id": "7", "name":
>>> > > "srv-5-comp.blabla.blub",
>>> > > >>>>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6",
>>> "type"="Routing"},
>>> > > Host
>>> > > >>>>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
>>> > > >>>>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
>>> > > {"id": "6",
>>> > > >>>>> "name": "srv-3-comp.blabla.blub", "uuid":
>>> > > >>>>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
>>> > > {"id": "1",
>>> > > >>>>> "name": "srv-2-comp.blabla.blub", "uuid":
>>> > > >>>>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
>>> > > >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for
>>> > > allocation
>>> > > >>>>> after prioritization: [Host {"id": "7", "name":
>>> > > "srv-5-comp.blabla.blub",
>>> > > >>>>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6",
>>> "type"="Routing"},
>>> > > Host
>>> > > >>>>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
>>> > > >>>>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
>>> > > {"id": "6",
>>> > > >>>>> "name": "srv-3-comp.blabla.blub", "uuid":
>>> > > >>>>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
>>> > > {"id": "1",
>>> > > >>>>> "name": "srv-2-comp.blabla.blub", "uuid":
>>> > > >>>>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
>>> > > >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for
>>> > speed=500Mhz,
>>> > > >>>>> Ram=1024 MB
>>> > > >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
>>> > > >>>>> srv-5-comp.blabla.blub, uuid:
>>> 8c753ce1-0253-49c9-9c7f-1f971a719ce6}
>>> > > is KVM
>>> > > >>>>> hypervisor type, no max guest limit check needed
>>> > > >>>>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu
>>> > capability
>>> > > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> > > speed: 500
>>> > > >>>>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7
>>> has
>>> > > enough
>>> > > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > > 1073741824 ,
>>> > > >>>>> cpuOverprovisioningFactor: 1.0
>>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> > CPU:
>>> > > 127200
>>> > > >>>>> and CPU after applying overprovisioning: 127200
>>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 ,
>>> > > Requested CPU:
>>> > > >>>>> 500
>>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37
>>> GB)
>>> > > >>>>> 189373743104 , Requested RAM: (1.00 GB) 1073741824
>>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
>>> and
>>> > > RAM
>>> > > >>>>> available
>>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> > from
>>> > > host:
>>> > > >>>>> 7, used: 90100, reserved: 0, actual total: 127200, total with
>>> > > >>>>> overprovisioning: 127200; requested
>>> > > cpu:500,alloc_from_last_host?:false
>>> > > >>>>> ,considerReservedCapacity?: true
>>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> > from
>>> > > host:
>>> > > >>>>> 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total:
>>> > > (250.27 GB)
>>> > > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > > alloc_from_last_host?:
>>> > > >>>>> false , considerReservedCapacity?: true
>>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable
>>> host,
>>> > > adding to
>>> > > >>>>> list: 7
>>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
>>> > > >>>>> srv-4-comp.blabla.blub, uuid:
>>> 134c625d-15ab-461a-a38d-36fb0b979b5d}
>>> > > is KVM
>>> > > >>>>> hypervisor type, no max guest limit check needed
>>> > > >>>>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu
>>> > capability
>>> > > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> > > speed: 500
>>> > > >>>>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5
>>> has
>>> > > enough
>>> > > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > > 1073741824 ,
>>> > > >>>>> cpuOverprovisioningFactor: 1.0
>>> > > >>>>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> > CPU:
>>> > > 127200
>>> > > >>>>> and CPU after applying overprovisioning: 127200
>>> > > >>>>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 ,
>>> > > Requested
>>> > > >>>>> CPU: 500
>>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77
>>> GB)
>>> > > >>>>> 251009040384 , Requested RAM: (1.00 GB) 1073741824
>>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
>>> and
>>> > > RAM
>>> > > >>>>> available
>>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> > from
>>> > > host:
>>> > > >>>>> 5, used: 11100, reserved: 0, actual total: 127200, total with
>>> > > >>>>> overprovisioning: 127200; requested
>>> > > cpu:500,alloc_from_last_host?:false
>>> > > >>>>> ,considerReservedCapacity?: true
>>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> > from
>>> > > host:
>>> > > >>>>> 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total:
>>> > > (250.27 GB)
>>> > > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > > alloc_from_last_host?:
>>> > > >>>>> false , considerReservedCapacity?: true
>>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable
>>> host,
>>> > > adding to
>>> > > >>>>> list: 5
>>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
>>> > > >>>>> srv-3-comp.blabla.blub, uuid:
>>> 6858713b-7e17-46d8-854d-24a13f9c7001}
>>> > > is KVM
>>> > > >>>>> hypervisor type, no max guest limit check needed
>>> > > >>>>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu
>>> > capability
>>> > > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> > > speed: 500
>>> > > >>>>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6
>>> has
>>> > > enough
>>> > > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > > 1073741824 ,
>>> > > >>>>> cpuOverprovisioningFactor: 1.0
>>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> > CPU:
>>> > > 127200
>>> > > >>>>> and CPU after applying overprovisioning: 127200
>>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 ,
>>> > > Requested CPU:
>>> > > >>>>> 500
>>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02
>>> GB)
>>> > > >>>>> 204032835584 , Requested RAM: (1.00 GB) 1073741824
>>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
>>> and
>>> > > RAM
>>> > > >>>>> available
>>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> > from
>>> > > host:
>>> > > >>>>> 6, used: 42900, reserved: 0, actual total: 127200, total with
>>> > > >>>>> overprovisioning: 127200; requested
>>> > > cpu:500,alloc_from_last_host?:false
>>> > > >>>>> ,considerReservedCapacity?: true
>>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> > from
>>> > > host:
>>> > > >>>>> 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total:
>>> > > (250.27 GB)
>>> > > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > > alloc_from_last_host?:
>>> > > >>>>> false , considerReservedCapacity?: true
>>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable
>>> host,
>>> > > adding to
>>> > > >>>>> list: 6
>>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
>>> > > >>>>> srv-2-comp.blabla.blub, uuid:
>>> 883f7f62-55f2-42f4-8242-16b6847c4204}
>>> > > is KVM
>>> > > >>>>> hypervisor type, no max guest limit check needed
>>> > > >>>>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu
>>> > capability
>>> > > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> > > speed: 500
>>> > > >>>>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1
>>> has
>>> > > enough
>>> > > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > > 1073741824 ,
>>> > > >>>>> cpuOverprovisioningFactor: 1.0
>>> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> > CPU:
>>> > > 127200
>>> > > >>>>> and CPU after applying overprovisioning: 127200
>>> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 ,
>>> > > Requested CPU:
>>> > > >>>>> 500
>>> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02
>>> GB)
>>> > > >>>>> 229802639360 , Requested RAM: (1.00 GB) 1073741824
>>> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
>>> and
>>> > > RAM
>>> > > >>>>> available
>>> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> > from
>>> > > host:
>>> > > >>>>> 1, used: 48200, reserved: 0, actual total: 127200, total with
>>> > > >>>>> overprovisioning: 127200; requested
>>> > > cpu:500,alloc_from_last_host?:false
>>> > > >>>>> ,considerReservedCapacity?: true
>>> > > >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> > from
>>> > > host:
>>> > > >>>>> 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total:
>>> > > (250.27 GB)
>>> > > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > > alloc_from_last_host?:
>>> > > >>>>> false , considerReservedCapacity?: true
>>> > > >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable
>>> host,
>>> > > adding to
>>> > > >>>>> list: 1
>>> > > >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator
>>> > returning 4
>>> > > >>>>> suitable hosts
>>> > > >>>>> 2022-08-29 10:51:56,260 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
>>> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
>>> state
>>> > > >>>>> transited from [Starting] to [Stopped] with event
>>> > [OperationFailed].
>>> > > VM's
>>> > > >>>>> original host: null, new host: null, host before state
>>> transition:
>>> > > null
>>> > > >>>>> 2022-08-29 10:51:56,260 WARN [c.c.v.VirtualMachineManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Unable to orchestrate start VM instance {id:
>>> "83",
>>> > > name:
>>> > > >>>>> "v-83-VM", uuid: "b1d1f9c6-eafe-4527-9335-e649c646aab0",
>>> > > >>>>> type="ConsoleProxy"} due to [Unable to create deployment, no
>>> usable
>>> > > volumes
>>> > > >>>>> found for the VM: 83].
>>> > > >>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to
>>> create
>>> > > >>>>> deployment, no usable volumes found for the VM: 83
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
>>> > > >>>>> at
>>> jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>>> > > >>>>> Source)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > > >>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>>> > > >>>>> at
>>> > > >>>>>
>>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> > > >>>>> at
>>> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> > > >>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>> > > >>>>> 2022-08-29 10:51:56,262 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Done executing VM work job:
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
>>> > > >>>>> 2022-08-29 10:51:56,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Complete async job-3539, jobStatus: FAILED,
>>> > > resultCode: 0,
>>> > > >>>>> result:
>>> > > >>>>>
>>> > >
>>> >
>>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODMiLCBuYW1lOiAidi04My1WTSIsIHV1aWQ6ICJiMWQxZjljNi1lYWZlLTQ1MjctOTMzNS1lNjQ5YzY0NmFhYjAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODNdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
>>> > > >>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Publish async job-3539 complete on message bus
>>> > > >>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Wake up jobs related to job-3539
>>> > > >>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Update db status for job-3539
>>> > > >>>>> 2022-08-29 10:51:56,266 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Wake up jobs joined with job-3539 and disjoin
>>> all
>>> > > subjobs
>>> > > >>>>> created from job- 3539
>>> > > >>>>> 2022-08-29 10:51:56,276 DEBUG [c.c.v.VmWorkJobDispatcher]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
>>> > > (logid:ba10fec5) Done
>>> > > >>>>> with run of VM work job: com.cloud.vm.VmWorkStart for VM 83,
>>> job
>>> > > origin:
>>> > > >>>>> 3445
>>> > > >>>>> 2022-08-29 10:51:56,276 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
>>> > > (logid:ba10fec5) Done
>>> > > >>>>> executing com.cloud.vm.VmWorkStart for job-3539
>>> > > >>>>> 2022-08-29 10:51:56,278 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
>>> > > (logid:ba10fec5)
>>> > > >>>>> Remove job-3539 from job monitoring
>>> > > >>>>> 2022-08-29 10:51:56,285 WARN [c.c.c.ConsoleProxyManagerImpl]
>>> > > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Unable to
>>> allocate
>>> > > console
>>> > > >>>>> proxy standby capacity for zone [1] due to [Unable to
>>> orchestrate
>>> > > start VM
>>> > > >>>>> instance {id: "83", name: "v-83-VM", uuid:
>>> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
>>> due to
>>> > > [Unable
>>> > > >>>>> to create deployment, no usable volumes found for the VM:
>>> 83].].
>>> > > >>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to
>>> > > orchestrate
>>> > > >>>>> start VM instance {id: "83", name: "v-83-VM", uuid:
>>> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
>>> due to
>>> > > [Unable
>>> > > >>>>> to create deployment, no usable volumes found for the VM: 83].
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
>>> > > >>>>> at
>>> jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>>> > > >>>>> Source)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > > >>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>>> > > >>>>> at
>>> > > >>>>>
>>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> > > >>>>> at
>>> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> > > >>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>> > > >>>>> 2022-08-29 10:51:56,287 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>>> > > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) received console
>>> > > proxy alert
>>> > > >>>>> 2022-08-29 10:51:56,288 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>>> > > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Console proxy
>>> > creation
>>> > > >>>>> failure, zone: blabla
>>> > > >>>>> 2022-08-29 10:51:56,289 WARN [c.c.a.AlertManagerImpl]
>>> > > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) alertType=[10]
>>> > > >>>>> dataCenterId=[1] podId=[null] clusterId=[null] message=[Console
>>> > proxy
>>> > > >>>>> creation failure. zone: blabla, error details: Unable to
>>> > orchestrate
>>> > > start
>>> > > >>>>> VM instance {id: "83", name: "v-83-VM", uuid:
>>> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
>>> due to
>>> > > [Unable
>>> > > >>>>> to create deployment, no usable volumes found for the VM:
>>> 83].].
>>> > > >>>>>
>>> > > >>>>>
>>> > > >>>>>
>>> > > >>>>>
>>> > > >>>>> Am Mo., 29. Aug. 2022 um 09:40 Uhr schrieb Slavka Peleva
>>> > > >>>>> <sl...@storpool.com.invalid>:
>>> > > >>>>>
>>> > > >>>>>> Hi Chris,
>>> > > >>>>>>
>>> > > >>>>>> Did you recreate the system VMs? In the 4.17 version, the
>>> > > >>>>>> `systemvm.iso` is
>>> > > >>>>>> deprecated. In its place, you should have the file on agents
>>> > > >>>>>> `/usr/share/cloudstack-common/vms/agent.zip`.
>>> > > >>>>>> Can you share the complete log if the system VMs are newly
>>> created
>>> > > >>>>>> after
>>> > > >>>>>> the upgrade?
>>> > > >>>>>>
>>> > > >>>>>> Best regards,
>>> > > >>>>>> Slavka
>>> > > >>>>>>
>>> > > >>>>>> On Sun, Aug 28, 2022 at 2:16 PM Vash_X@gmx.de <Va...@gmx.de>
>>> > > wrote:
>>> > > >>>>>>
>>> > > >>>>>> > Hi everyone,
>>> > > >>>>>> >
>>> > > >>>>>> > faceing some challanges again after upgradeing to 4.17.
>>> > > >>>>>> > Did as explained in the docs.
>>> > > >>>>>> >
>>> > > >>>>>> > Afterwards I am now not able to migrate system VMs to
>>> different
>>> > > >>>>>> hosts.
>>> > > >>>>>> > Getting the error
>>> > > >>>>>> > "Exception during migrate: org.libvirt.LibvirtException:
>>> Cannot
>>> > > >>>>>> access
>>> > > >>>>>> > storage file
>>> '/usr/share/cloudstack-common/vms/systemvm.iso': No
>>> > > >>>>>> such file
>>> > > >>>>>> > or directory"
>>> > > >>>>>> >
>>> > > >>>>>> > Checked the exitstence of the file on the servers:
>>> > > >>>>>> > Mangement-Server has a "systemvm.iso.bak"
>>> > > >>>>>> > Hosts - no "systemvm.iso" at all present.
>>> > > >>>>>> >
>>> > > >>>>>> > Maybe someone has an idea on how to move on with this?
>>> > > >>>>>> >
>>> > > >>>>>> > Regards,
>>> > > >>>>>> > Chris
>>> > > >>>>>> >
>>> > > >>>>>>
>>> > > >>>>>
>>> > >
>>> >
>>>
>>
Re: Error migrate System VMs after upgrageing to 4.17
Posted by "Vash_X@gmx.de" <Va...@gmx.de>.
Hi,
time for a short update.
Today i tried my luck by switching the global setting
*system.vm.public.ip.reservation.mode.strictness* from true to false.
Afterwards i was able to create a systemvm and migrate it to another
server. --> @Slavka Peleva <sl...@storpool.com> so the initial "problem"
of not finding systemvm.iso seems to be solved.
The used public IP "belongs" to a ip range reserved for the domain "Root" (
[Root] in WebGUI Style) which is not marked for use with systemVMs.
As much as i like to have the system up running again, still the question
is why the reservation marked for usage of systemVMs - belonging to the
account system as the GUI says - isn't used anymore?
PS.:
I tried to create a new, additional ip range wich is marked as "Reserved
for SystemVMs" for the domain root. Afterwards I enabled the global setting
as described to "true".
Afterwards i distroyed a consoleproxy. The deployment of the new proxy
fails due to "insufficent address capacity". So it seems the IP Range is
not "respected"?
I checked the table "domain_vlan_map", which is used to get the vlan
information and there is NO entry of the newly created ip range...
regards
Christopher
Am Do., 1. Sept. 2022 um 01:01 Uhr schrieb Vash_X@gmx.de <Va...@gmx.de>:
> Hi Slavka,
>
> be prepared vor a longer answer as this seems to be a more specific
> problem - but first let me thank you for your guidance and support!
>
> > The discussion started about a problem migrating the system VMs, and
> probably many other issues will appear.
> Don't give up :-D I might see some light ahead....
>
> Bevore getting into the logs and my findings let me answer you your
> questions briefly.
> > Can you tell from which version you started the upgrade?
> I started from 4.16.1; the installation was builded with 4.15.1 and then
> updated to 4.16 / 4.16.1
> > Did the upgrade was successful for the management/agents?
> The upgrade was successfull - even if we had our issues which are
> currently in a ticket at github
> https://github.com/apache/cloudstack/issues/6680
> The upgrade of the management server was successfully performed too.
> > Did you restart the agents on all machines?
> We did as we had trouble with the changed detection of CPU speeds on the
> host.
> The whole "problem" of the SystemVM.iso is at least from my point of view
> no problem of the agent-update.
> a) we had the expection from reading the upgradeguide, that a "simple"
> patching of the systemVMs would be enough. That at least was our
> understanding of the "matrix" showing the patching capabilities of the
> different versions / "upgrade paths".
> b) currently we assume, that the main-reason for the failing migration is,
> that the SSVM / CPVM are still running without a clear "restart". Which we
> can't test as we still have to solve the problem of the public IP
> allocation.
> > Can you share the content in the directory
> `/usr/share/cloudstack-agent/lib/` from the hypervisor where the migration
> failed?
>
> activation-1.1.1.jar
> cloud-framework-events-4.17.0.1.jar
> error_prone_annotations-2.3.4.jar jaxb-impl-2.3.0.jar
> okio-2.8.0.jar
> annotations-13.0.jar
> cloud-framework-ipc-4.17.0.1.jar esapi-2.1.0.1.jar
> jbcrypt-1.0.0.jar
> opensaml-2.6.6.jar
> antisamy-1.5.3.jar
> cloud-framework-jobs-4.17.0.1.jar failureaccess-1.0.1.jar
> jcl-over-slf4j-1.7.5.jar
> openws-1.5.6.jar
> aopalliance-repackaged-2.5.0-b42.jar
> cloud-framework-managed-context-4.17.0.1.jar groovy-all-2.4.17.jar
> jersey-client-2.26.jar
> osgi-resource-locator-1.0.1.jar
> apache-log4j-extras-1.2.17.jar
> cloud-framework-rest-4.17.0.1.jar gson-1.7.2.jar
> jersey-common-2.26.jar
> protobuf-java-3.11.1.jar
> aspectjweaver-1.8.13.jar
> cloud-framework-security-4.17.0.1.jar guava-30.1-jre.jar
> jersey-entity-filtering-2.26.jar
> rados-0.6.0.jar
> aws-java-sdk-core-1.11.940.jar
> cloud-plugin-hypervisor-kvm-4.17.0.1.jar guava-testlib-18.0.jar
> jersey-hk2-2.26.jar
> reflections-0.9.12.jar
> aws-java-sdk-kms-1.11.940.jar
> cloud-plugin-network-ovs-4.17.0.1.jar hk2-api-2.5.0-b42.jar
> jersey-media-json-jackson-2.26.jar
> reload4j-1.2.20.jar
> aws-java-sdk-s3-1.11.940.jar
> cloud-plugin-storage-volume-scaleio-4.17.0.1.jar hk2-locator-2.5.0-b42.jar
> jersey-media-multipart-2.26.jar
> retrofit-2.9.0.jar
> batik-constants-1.14.jar
> cloud-secondary-storage-4.17.0.1.jar hk2-utils-2.5.0-b42.jar
> jmespath-java-1.11.940.jar
> rolling-maintenance
> batik-css-1.14.jar cloud-server-4.17.0.1.jar
> httpclient-4.5.13.jar
> jna-5.5.0.jar
> slf4j-api-1.7.5.jar
> batik-i18n-1.14.jar cloud-utils-4.17.0.1.jar
> httpcore-4.4.14.jar
> jna-platform-5.5.0.jar
> spring-aop-5.3.18.jar
> batik-shared-resources-1.14.jar commons-beanutils-1.9.2.jar
> influxdb-java-2.21.jar
> joda-time-2.10.9.jar
> spring-beans-5.3.18.jar
> batik-util-1.14.jar commons-codec-1.15.jar
> ini4j-0.5.4.jar
> jsch-0.1.55.jar
> spring-context-5.3.18.jar
> bcpkix-jdk15on-1.70.jar
> commons-collections-3.2.1.jar ion-java-1.0.2.jar
> json-20090211.jar
> spring-core-5.3.18.jar
> bcprov-jdk15on-1.70.jar commons-compress-1.21.jar
> j2objc-annotations-1.3.jar
> jsr305-1.3.9.jar
> spring-expression-5.3.18.jar
> bctls-jdk15on-1.70.jar
> commons-configuration-1.10.jar
> jackson-annotations-2.12.5.jar jul-to-slf4j-1.7.5.jar
> spring-jcl-5.3.18.jar
> bcutil-jdk15on-1.70.jar commons-daemon-1.2.3.jar
> jackson-core-2.12.5.jar
> kotlin-stdlib-1.4.10.jar
> spring-web-5.3.18.jar
> bsh-core-2.0b4.jar commons-dbcp2-2.8.0.jar
> jackson-databind-2.12.5.jar
> kotlin-stdlib-common-1.4.0.jar
> stax2-api-3.1.4.jar
> cglib-nodep-3.3.0.jar commons-digester-1.8.1.jar
> jackson-dataformat-cbor-2.6.7.jar
> kotlin-stdlib-jdk7-1.4.10.jar
> swagger-annotations-2.0.0.jar
> checker-qual-3.5.0.jar commons-email-1.5.jar
> jackson-datatype-jsr310-2.12.5.jar
> kotlin-stdlib-jdk8-1.4.10.jar
> tink-1.5.0.jar
> cloud-agent-4.17.0.1.jar commons-fileupload-1.4.jar
> jackson-jaxrs-base-2.12.1.jar
> libvirt-0.5.2.jar
> trilead-ssh2-build-217-jenkins-27.jar
> cloud-api-4.17.0.1.jar commons-httpclient-3.1.jar
> jackson-jaxrs-json-provider-2.12.1.jar
> libvirtqemuhook
> truth-0.23.jar
> cloud-core-4.17.0.1.jar commons-io-2.8.0.jar
> jackson-module-jaxb-annotations-2.8.4.jar
> listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
> velocity-1.7.jar
> cloud-engine-api-4.17.0.1.jar commons-lang-2.6.jar
> jasypt-1.9.3.jar
> log4j-over-slf4j-1.7.5.jar
> woodstox-core-5.0.3.jar
> cloud-engine-components-api-4.17.0.1.jar commons-lang3-3.11.jar
> java-ipv6-0.17.jar
> logging-interceptor-4.9.0.jar
> xalan-2.7.0.jar
> cloud-engine-schema-4.17.0.1.jar commons-logging-1.2.jar
> java-linstor-0.3.0.jar
> mail-1.5.0-b01.jar
> xercesImpl-2.8.0.jar
> cloud-engine-storage-4.17.0.1.jar commons-math3-3.6.1.jar
> javassist-3.27.0-GA.jar
> metrics-core-3.0.2.jar
> xmlgraphics-commons-2.6.jar
> cloud-engine-storage-configdrive-4.17.0.1.jar commons-net-3.7.2.jar
> javax.annotation-api-1.3.2.jar
> metrics-jvm-3.0.2.jar
> xmlpull-1.1.3.1.jar
> cloud-engine-storage-image-4.17.0.1.jar commons-pool2-2.9.0.jar
> javax.inject-1.jar
> mimepull-1.9.6.jar
> xmlschema-core-2.2.5.jar
> cloud-engine-storage-volume-4.17.0.1.jar commons-validator-1.6.jar
> javax.inject-2.5.0-b42.jar
> moshi-1.8.0.jar
> xmlsec-1.5.7.jar
> cloud-framework-agent-lb-4.17.0.1.jar converter-moshi-2.9.0.jar
> javax.mail-1.5.6.jar
> msgpack-core-0.8.21.jar
> xmltooling-1.4.6.jar
> cloud-framework-ca-4.17.0.1.jar cxf-core-3.2.14.jar
> javax.persistence-2.2.1.jar
> mxparser-1.2.2.jar
> xom-1.2.5.jar
> cloud-framework-cluster-4.17.0.1.jar
> cxf-rt-frontend-jaxrs-3.2.14.jar
> javax.servlet-api-4.0.1.jar nekohtml-1.9.16.jar
> xstream-1.4.19.jar
> cloud-framework-config-4.17.0.1.jar
> cxf-rt-transports-http-3.2.14.jar javax.ws.rs-api-2.1.jar
> netty-all-4.1.42.Final.jar
> cloud-framework-db-4.17.0.1.jar eddsa-0.3.0.jar
> jaxb-api-2.3.0.jar
> not-yet-commons-ssl-0.3.9.jar
> cloud-framework-direct-download-4.17.0.1.jar ehcache-core-2.6.11.jar
> jaxb-core-2.3.0.jar
>
> And now to the problem of the ip's....
> Thanks for your guidance, i guess i can give some more information. For
> the findings to understand i guess i need to elaborate a bit on the current
> configuration.
>
> Currently we are having only one Zone up and running (ID=1).
> In this Zone we have 3 different domains which are "Root","A","B".
> Each domain has a account linked: Root = Admin / A = AccA / B = AccB.
> The public network of the zone has currently 3 ip ranges which are: [Root]
> system (System VMs = true) / [Root] / [B]AccB (written like in the WebGUI
> which is i guess going to be important for understanding).
> This kind of configuration has been in place since the initial setup and
> has been working without problems.
>
> Afterwards you will find the logs with the database querys - i started
> with a deleting the stopped consoleproxy. i will only share the part when
> the process of configuring the NIC / IPs for the console proxy will take
> place.
> I put some statements into it - for me to get a better understanding of
> the process - and i hope that you will get through this a bit better. The
> comments are referenceing on the following "closing" statements. How ever i
> didn't deleted anything.
>
>
> ###################################################################### Nic
> allocation
>
> 2022-08-31 18:44:21,203 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Allocating nics for VM
> instance {id: "86", name: "v-86-VM", uuid:
> "6f77acb4-c65e-49e8-ae03-1175b3974412", type="ConsoleProxy"}
> 2022-08-31 18:44:21,203 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,204 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: start requested by:
> -Transaction.execute:39-NetworkOrchestrator.allocate:778-VirtualMachineManagerImpl$1.doInTransactionWithoutResult:478-TransactionCallbackWithExceptionNoReturn.doInTransaction:25-TransactionCallbackWithExceptionNoReturn.doInTransaction:21-Transaction.execute:40-VirtualMachineManagerImpl.allocate:469-VirtualMachineManagerImpl.allocate:528-ConsoleProxyManagerImpl.createProxyInstance:700-ConsoleProxyManagerImpl.startNew:572-ConsoleProxyManagerImpl.allocCapacity:809-ConsoleProxyManagerImpl.expandPool:1537
> 2022-08-31 18:44:21,204 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: has already been
> started.
> 2022-08-31 18:44:21,204 TRACE [o.a.c.e.o.NetworkOrchestrator]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) allocating networks for
> v-86-VM(template c20799e4-a990-449a-aae6-4b67374c7aff); 3 networks
> 2022-08-31 18:44:21,204 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,206 TRACE [c.c.u.d.T.Connection]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Creating a DB connection
> with txn: for 0: dbconn623913035. Stack:
> -TransactionLegacy.prepareStatement:466-TransactionLegacy.prepareAutoCloseStatement:459-GenericDaoBase.searchIncludingRemoved:396-GenericDaoBase.searchIncludingRemoved:359-GenericDaoBase.search:348-GenericDaoBase.search:1319-TemplateDeployAsIsDetailsDaoImpl.listDetailsByTemplateIdMatchingPrefix:64-TemplateDeployAsIsDetailsDaoImpl.listNetworkRequirementsByTemplateId:69-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62-DelegatingMethodAccessorImpl.invoke:43-Method.invoke:566
> 2022-08-31 18:44:21,206 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> template_deploy_as_is_details.id,
> template_deploy_as_is_details.template_id,
> template_deploy_as_is_details.name, template_deploy_as_is_details.value
> FROM template_deploy_as_is_details WHERE
> template_deploy_as_is_details.template_id = ? AND
> template_deploy_as_is_details.name LIKE ?
>
> #### template_deploy_as_is_details = empty --> no results returned
>
> 2022-08-31 18:44:21,206 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT
> template_deploy_as_is_details.id,
> template_deploy_as_is_details.template_id,
> template_deploy_as_is_details.name, template_deploy_as_is_details.value
> FROM template_deploy_as_is_details WHERE
> template_deploy_as_is_details.template_id = 213 AND
> template_deploy_as_is_details.name LIKE x'6E6574776F726B2D25'
> 2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Connection]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: Not closing DB
> connection because we're still in a transaction.
> 2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> networks.id, networks.mode, networks.broadcast_domain_type,
> networks.traffic_type, networks.name, networks.display_text,
> networks.broadcast_uri, networks.gateway, networks.cidr,
> networks.network_cidr, networks.network_offering_id, networks.vpc_id,
> networks.physical_network_id, networks.data_center_id, networks.related,
> networks.guru_name, networks.state, networks.redundant, networks.dns1,
> networks.domain_id, networks.account_id, networks.set_fields,
> networks.guru_data, networks.dns2, networks.network_domain,
> networks.removed, networks.created, networks.reservation_id, networks.uuid,
> networks.guest_type, networks.acl_type, networks.restart_required,
> networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr,
> networks.display_network, networks.network_acl_id, networks.streched_l2,
> networks.external_id FROM networks WHERE networks.id = ? AND
> networks.removed IS NULL
> 2022-08-31 18:44:21,209 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Allocating nic for vm VM
> instance {id: "86", name: "v-86-VM", uuid:
> "6f77acb4-c65e-49e8-ae03-1175b3974412", type="ConsoleProxy"} in network
> Network {"id": 200, "name": "null", "uuid":
> "1b0bf965-a391-478d-833d-3e37e4a15eb8", "networkofferingid": 1} with
> requested profile NicProfile
> {"broadcastUri":null,"iPv4Address":null,"id":0,"reservationId":null,"vmId":0}
> 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> data_center.id, data_center.name, data_center.description,
> data_center.dns1, data_center.dns2, data_center.ip6_dns1,
> data_center.ip6_dns2, data_center.internal_dns1, data_center.internal_dns2,
> data_center.router_mac_address, data_center.guest_network_cidr,
> data_center.domain_id, data_center.domain, data_center.networktype,
> data_center.dns_provider, data_center.dhcp_provider,
> data_center.gateway_provider, data_center.vpn_provider,
> data_center.userdata_provider, data_center.lb_provider,
> data_center.firewall_provider, data_center.mac_address,
> data_center.zone_token, data_center.removed, data_center.allocation_state,
> data_center.uuid, data_center.is_security_group_enabled,
> data_center.is_local_storage_enabled, data_center.sort_key FROM data_center
> WHERE data_center.id = ? AND data_center.removed IS NULL
>
> ### Allocating public nic
>
> 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT networks.id,
> networks.mode, networks.broadcast_domain_type, networks.traffic_type,
> networks.name, networks.display_text, networks.broadcast_uri,
> networks.gateway, networks.cidr, networks.network_cidr,
> networks.network_offering_id, networks.vpc_id,
> networks.physical_network_id, networks.data_center_id, networks.related,
> networks.guru_name, networks.state, networks.redundant, networks.dns1,
> networks.domain_id, networks.account_id, networks.set_fields,
> networks.guru_data, networks.dns2, networks.network_domain,
> networks.removed, networks.created, networks.reservation_id, networks.uuid,
> networks.guest_type, networks.acl_type, networks.restart_required,
> networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr,
> networks.display_network, networks.network_acl_id, networks.streched_l2,
> networks.external_id FROM networks WHERE networks.id = 200 AND
> networks.removed IS NULL
> 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> account.id, account.account_name, account.type, account.role_id,
> account.domain_id, account.state, account.created, account.removed,
> account.cleanup_needed, account.network_domain, account.uuid,
> account.default_zone_id, account.default FROM account WHERE account.id =
> ? AND account.removed IS NULL
>
> ### Getting Zone Information
>
> 2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT data_center.id,
> data_center.name, data_center.description, data_center.dns1,
> data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2,
> data_center.internal_dns1, data_center.internal_dns2,
> data_center.router_mac_address, data_center.guest_network_cidr,
> data_center.domain_id, data_center.domain, data_center.networktype,
> data_center.dns_provider, data_center.dhcp_provider,
> data_center.gateway_provider, data_center.vpn_provider,
> data_center.userdata_provider, data_center.lb_provider,
> data_center.firewall_provider, data_center.mac_address,
> data_center.zone_token, data_center.removed, data_center.allocation_state,
> data_center.uuid, data_center.is_security_group_enabled,
> data_center.is_local_storage_enabled, data_center.sort_key FROM data_center
> WHERE data_center.id = 1 AND data_center.removed IS NULL
> 2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: start requested by:
> -Transaction.execute:39-IpAddressManagerImpl.listAvailablePublicIps:824-IpAddressManagerImpl.fetchNewPublicIp:811-IpAddressManagerImpl.fetchNewPublicIp:803-IpAddressManagerImpl.assignPublicIpAddress:784-PublicNetworkGuru.getIp:125-PublicNetworkGuru.allocate:172-NetworkOrchestrator.allocateNic:973-NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice:823-NetworkOrchestrator$3.doInTransactionWithoutResult:797-TransactionCallbackWithExceptionNoReturn.doInTransaction:25-TransactionCallbackWithExceptionNoReturn.doInTransaction:21
> 2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: has already been
> started.
> 2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> data_center.id, data_center.name, data_center.description,
> data_center.dns1, data_center.dns2, data_center.ip6_dns1,
> data_center.ip6_dns2, data_center.internal_dns1, data_center.internal_dns2,
> data_center.router_mac_address, data_center.guest_network_cidr,
> data_center.domain_id, data_center.domain, data_center.networktype,
> data_center.dns_provider, data_center.dhcp_provider,
> data_center.gateway_provider, data_center.vpn_provider,
> data_center.userdata_provider, data_center.lb_provider,
> data_center.firewall_provider, data_center.mac_address,
> data_center.zone_token, data_center.removed, data_center.allocation_state,
> data_center.uuid, data_center.is_security_group_enabled,
> data_center.is_local_storage_enabled, data_center.sort_key FROM data_center
> WHERE data_center.id = ? AND data_center.removed IS NULL
>
> ### Account Information "system"
>
> 2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT account.id,
> account.account_name, account.type, account.role_id, account.domain_id,
> account.state, account.created, account.removed, account.cleanup_needed,
> account.network_domain, account.uuid, account.default_zone_id,
> account.default FROM account WHERE account.id = 1 AND account.removed IS
> NULL
> 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> networks.id, networks.mode, networks.broadcast_domain_type,
> networks.traffic_type, networks.name, networks.display_text,
> networks.broadcast_uri, networks.gateway, networks.cidr,
> networks.network_cidr, networks.network_offering_id, networks.vpc_id,
> networks.physical_network_id, networks.data_center_id, networks.related,
> networks.guru_name, networks.state, networks.redundant, networks.dns1,
> networks.domain_id, networks.account_id, networks.set_fields,
> networks.guru_data, networks.dns2, networks.network_domain,
> networks.removed, networks.created, networks.reservation_id, networks.uuid,
> networks.guest_type, networks.acl_type, networks.restart_required,
> networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr,
> networks.display_network, networks.network_acl_id, networks.streched_l2,
> networks.external_id FROM networks WHERE networks.id = ? AND
> networks.removed IS NULL
>
> ### Getting Zone Information
>
> 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT data_center.id,
> data_center.name, data_center.description, data_center.dns1,
> data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2,
> data_center.internal_dns1, data_center.internal_dns2,
> data_center.router_mac_address, data_center.guest_network_cidr,
> data_center.domain_id, data_center.domain, data_center.networktype,
> data_center.dns_provider, data_center.dhcp_provider,
> data_center.gateway_provider, data_center.vpn_provider,
> data_center.userdata_provider, data_center.lb_provider,
> data_center.firewall_provider, data_center.mac_address,
> data_center.zone_token, data_center.removed, data_center.allocation_state,
> data_center.uuid, data_center.is_security_group_enabled,
> data_center.is_local_storage_enabled, data_center.sort_key FROM data_center
> WHERE data_center.id = 1 AND data_center.removed IS NULL
> 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> account_vlan_map.id, account_vlan_map.account_id,
> account_vlan_map.vlan_db_id FROM account_vlan_map WHERE
> account_vlan_map.account_id = ?
>
> ### Getting network information
>
> 2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT networks.id,
> networks.mode, networks.broadcast_domain_type, networks.traffic_type,
> networks.name, networks.display_text, networks.broadcast_uri,
> networks.gateway, networks.cidr, networks.network_cidr,
> networks.network_offering_id, networks.vpc_id,
> networks.physical_network_id, networks.data_center_id, networks.related,
> networks.guru_name, networks.state, networks.redundant, networks.dns1,
> networks.domain_id, networks.account_id, networks.set_fields,
> networks.guru_data, networks.dns2, networks.network_domain,
> networks.removed, networks.created, networks.reservation_id, networks.uuid,
> networks.guest_type, networks.acl_type, networks.restart_required,
> networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr,
> networks.display_network, networks.network_acl_id, networks.streched_l2,
> networks.external_id FROM networks WHERE networks.id = null AND
> networks.removed IS NULL
> 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> domain_vlan_map.id, domain_vlan_map.domain_id, domain_vlan_map.vlan_db_id
> FROM domain_vlan_map WHERE domain_vlan_map.domain_id = ?
>
> #### Account_vlan_map - no entry for account "System" nor for Account
> "Admin"
>
> 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT account_vlan_map.id,
> account_vlan_map.account_id, account_vlan_map.vlan_db_id FROM
> account_vlan_map WHERE account_vlan_map.account_id = 1
> 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT vlan.id,
> vlan.vlan_id, vlan.vlan_gateway, vlan.vlan_netmask, vlan.ip6_gateway,
> vlan.ip6_cidr, vlan.data_center_id, vlan.ip4_range, vlan.ip6_range,
> vlan.network_id, vlan.physical_network_id, vlan.vlan_type, vlan.uuid,
> vlan.removed, vlan.created FROM vlan LEFT OUTER JOIN domain_vlan_map ON
> vlan.id=domain_vlan_map.vlan_db_id LEFT OUTER JOIN account_vlan_map ON
> vlan.id=account_vlan_map.vlan_db_id WHERE vlan.data_center_id = ? AND
> vlan.removed IS NULL AND (domain_vlan_map.domain_id IS NULL ) AND
> (account_vlan_map.account_id IS NULL )
>
> ### Checking for availeable VLANS wich are NOT assigned to ANY Account NOR
> to ANY Domain --> Result is the intended vlan with expected IP Range
>
> 2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT domain_vlan_map.id,
> domain_vlan_map.domain_id, domain_vlan_map.vlan_db_id FROM domain_vlan_map
> WHERE domain_vlan_map.domain_id = 1
> 2022-08-31 18:44:21,214 TRACE [c.c.u.d.GenericDaoBase]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) join search statement is
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT vlan.id, vlan.vlan_id,
> vlan.vlan_gateway, vlan.vlan_netmask, vlan.ip6_gateway, vlan.ip6_cidr,
> vlan.data_center_id, vlan.ip4_range, vlan.ip6_range, vlan.network_id,
> vlan.physical_network_id, vlan.vlan_type, vlan.uuid, vlan.removed,
> vlan.created FROM vlan LEFT OUTER JOIN domain_vlan_map ON vlan.id=domain_vlan_map.vlan_db_id
> LEFT OUTER JOIN account_vlan_map ON vlan.id=account_vlan_map.vlan_db_id
> WHERE vlan.data_center_id = 1 AND vlan.removed IS NULL AND
> (domain_vlan_map.domain_id IS NULL ) AND (account_vlan_map.account_id IS
> NULL )
> 2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> configuration.instance, configuration.component, configuration.name,
> configuration.value, configuration.default_value,
> configuration.description, configuration.category,
> configuration.is_dynamic, configuration.scope, configuration.updated FROM
> configuration WHERE configuration.name = ?
>
> ### Getting further information on the vlan --> correct choice of expected
> VLAN + IP-Range
>
> 2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT vlan.id, vlan.vlan_id,
> vlan.vlan_gateway, vlan.vlan_netmask, vlan.ip6_gateway, vlan.ip6_cidr,
> vlan.data_center_id, vlan.ip4_range, vlan.ip6_range, vlan.network_id,
> vlan.physical_network_id, vlan.vlan_type, vlan.uuid, vlan.removed,
> vlan.created FROM vlan LEFT OUTER JOIN domain_vlan_map ON vlan.id=domain_vlan_map.vlan_db_id
> LEFT OUTER JOIN account_vlan_map ON vlan.id=account_vlan_map.vlan_db_id
> WHERE vlan.data_center_id = 1 AND vlan.removed IS NULL AND
> (domain_vlan_map.domain_id IS NULL ) AND (account_vlan_map.account_id IS
> NULL )
> 2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> user_ip_address.id, user_ip_address.account_id,
> user_ip_address.domain_id, user_ip_address.public_ip_address,
> user_ip_address.data_center_id, user_ip_address.source_nat,
> user_ip_address.allocated, user_ip_address.vlan_db_id,
> user_ip_address.one_to_one_nat, user_ip_address.vm_id,
> user_ip_address.state, user_ip_address.mac_address,
> user_ip_address.source_network_id, user_ip_address.network_id,
> user_ip_address.uuid, user_ip_address.physical_network_id,
> user_ip_address.is_system, user_ip_address.vpc_id,
> user_ip_address.dnat_vmip, user_ip_address.is_portable,
> user_ip_address.display, user_ip_address.rule_state,
> user_ip_address.forsystemvms, user_ip_address.removed,
> user_ip_address.created FROM user_ip_address INNER JOIN vlan ON
> user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id =
> ? AND user_ip_address.allocated IS NULL AND user_ip_address.vlan_db_id IN
> (?,?,?,?) AND user_ip_address.forsystemvms = ? AND
> user_ip_address.removed IS NULL AND (vlan.vlan_type = ? ) ORDER BY
> user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id ASC LIMIT
> 0, 1 FOR UPDATE
>
> ### Check of setting system.vm.public.ip.reservation.mode.strictness -->
> value "true"
>
> 2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT configuration.instance,
> configuration.component, configuration.name, configuration.value,
> configuration.default_value, configuration.description,
> configuration.category, configuration.is_dynamic, configuration.scope,
> configuration.updated FROM configuration WHERE configuration.name =
> x'73797374656D2E766D2E7075626C69632E69702E7265736572766174696F6E2E6D6F64652E7374726963746E657373'
>
> 2022-08-31 18:44:21,215 TRACE [c.c.u.d.GenericDaoBase]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) join search statement is
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT user_ip_address.id,
> user_ip_address.account_id, user_ip_address.domain_id,
> user_ip_address.public_ip_address, user_ip_address.data_center_id,
> user_ip_address.source_nat, user_ip_address.allocated,
> user_ip_address.vlan_db_id, user_ip_address.one_to_one_nat,
> user_ip_address.vm_id, user_ip_address.state, user_ip_address.mac_address,
> user_ip_address.source_network_id, user_ip_address.network_id,
> user_ip_address.uuid, user_ip_address.physical_network_id,
> user_ip_address.is_system, user_ip_address.vpc_id,
> user_ip_address.dnat_vmip, user_ip_address.is_portable,
> user_ip_address.display, user_ip_address.rule_state,
> user_ip_address.forsystemvms, user_ip_address.removed,
> user_ip_address.created FROM user_ip_address INNER JOIN vlan ON
> user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id =
> 1 AND user_ip_address.allocated IS NULL AND user_ip_address.vlan_db_id IN
> (3,10,13,14) AND user_ip_address.forsystemvms = 1 AND
> user_ip_address.removed IS NULL AND (vlan.vlan_type = 'VirtualNetwork' )
> ORDER BY user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id ASC
> LIMIT 0, 1 FOR UPDATE
> 2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> account_details.id, account_details.account_id, account_details.name,
> account_details.value FROM account_details WHERE account_details.account_id
> = ? AND account_details.name = ? ORDER BY RAND() LIMIT 1
>
> ### Trying to fetch IP-Address: No value / ip address returned - Domain
> "Root" shall provide ip addresses; thats where the vlanIDs are comming from
> --> user_ip_address.vlan_db_id IN (3,10,13,14)
>
> 2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT user_ip_address.id,
> user_ip_address.account_id, user_ip_address.domain_id,
> user_ip_address.public_ip_address, user_ip_address.data_center_id,
> user_ip_address.source_nat, user_ip_address.allocated,
> user_ip_address.vlan_db_id, user_ip_address.one_to_one_nat,
> user_ip_address.vm_id, user_ip_address.state, user_ip_address.mac_address,
> user_ip_address.source_network_id, user_ip_address.network_id,
> user_ip_address.uuid, user_ip_address.physical_network_id,
> user_ip_address.is_system, user_ip_address.vpc_id,
> user_ip_address.dnat_vmip, user_ip_address.is_portable,
> user_ip_address.display, user_ip_address.rule_state,
> user_ip_address.forsystemvms, user_ip_address.removed,
> user_ip_address.created FROM user_ip_address INNER JOIN vlan ON
> user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id =
> 1 AND user_ip_address.allocated IS NULL AND user_ip_address.vlan_db_id IN
> (3,10,13,14) AND user_ip_address.forsystemvms = 1 AND
> user_ip_address.removed IS NULL AND (vlan.vlan_type = 'VirtualNetwork' )
> ORDER BY user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id ASC
> LIMIT 0, 1 FOR UPDATE
> 2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> configuration.instance, configuration.component, configuration.name,
> configuration.value, configuration.default_value,
> configuration.description, configuration.category,
> configuration.is_dynamic, configuration.scope, configuration.updated FROM
> configuration WHERE configuration.name = ? ORDER BY RAND() LIMIT 1
>
> ### Trying to get information from the table account_detail for the
> account "system". Table is empty so no result back
>
> 2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT account_details.id,
> account_details.account_id, account_details.name, account_details.value
> FROM account_details WHERE account_details.account_id = 1 AND
> account_details.name = x'7573652E73797374656D2E7075626C69632E697073'
> ORDER BY RAND() LIMIT 1
> 2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> configuration.instance, configuration.component, configuration.name,
> configuration.value, configuration.default_value,
> configuration.description, configuration.category,
> configuration.is_dynamic, configuration.scope, configuration.updated FROM
> configuration WHERE configuration.name = ?
>
> ### Check for Configuration enable.account.settings.for.domain --> current
> value = false
>
> 2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT configuration.instance,
> configuration.component, configuration.name, configuration.value,
> configuration.default_value, configuration.description,
> configuration.category, configuration.is_dynamic, configuration.scope,
> configuration.updated FROM configuration WHERE configuration.name =
> x'656E61626C652E6163636F756E742E73657474696E67732E666F722E646F6D61696E'
> ORDER BY RAND() LIMIT 1
> 2022-08-31 18:44:21,217 WARN [c.c.n.IpAddressManagerImpl]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Unable to get ip address in
> zone id=1, vlanId id=[3, 10, 13, 14]
> 2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
> consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
> 2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
> connection
> 2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
> data_center.id, data_center.name, data_center.description,
> data_center.dns1, data_center.dns2, data_center.ip6_dns1,
> data_center.ip6_dns2, data_center.internal_dns1, data_center.internal_dns2,
> data_center.router_mac_address, data_center.guest_network_cidr,
> data_center.domain_id, data_center.domain, data_center.networktype,
> data_center.dns_provider, data_center.dhcp_provider,
> data_center.gateway_provider, data_center.vpn_provider,
> data_center.userdata_provider, data_center.lb_provider,
> data_center.firewall_provider, data_center.mac_address,
> data_center.zone_token, data_center.removed, data_center.allocation_state,
> data_center.uuid, data_center.is_security_group_enabled,
> data_center.is_local_storage_enabled, data_center.sort_key FROM data_center
> WHERE data_center.id = ? AND data_center.removed IS NULL
>
> ### Check for setting use.system.public.ips --> current value = false
>
> 2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT configuration.instance,
> configuration.component, configuration.name, configuration.value,
> configuration.default_value, configuration.description,
> configuration.category, configuration.is_dynamic, configuration.scope,
> configuration.updated FROM configuration WHERE configuration.name =
> x'7573652E73797374656D2E7075626C69632E697073'
>
> ### getting information for the current data_center / zone
>
> 2022-08-31 18:44:21,219 TRACE [c.c.u.d.T.Statement]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT data_center.id,
> data_center.name, data_center.description, data_center.dns1,
> data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2,
> data_center.internal_dns1, data_center.internal_dns2,
> data_center.router_mac_address, data_center.guest_network_cidr,
> data_center.domain_id, data_center.domain, data_center.networktype,
> data_center.dns_provider, data_center.dhcp_provider,
> data_center.gateway_provider, data_center.vpn_provider,
> data_center.userdata_provider, data_center.lb_provider,
> data_center.firewall_provider, data_center.mac_address,
> data_center.zone_token, data_center.removed, data_center.allocation_state,
> data_center.uuid, data_center.is_security_group_enabled,
> data_center.is_local_storage_enabled, data_center.sort_key FROM data_center
> WHERE data_center.id = 1 AND data_center.removed IS NULL
> 2022-08-31 18:44:21,219 DEBUG [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Rolling back the
> transaction: Time = 17 Name = consoleproxy-1; called by
> -TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-Transaction.execute:38-IpAddressManagerImpl.listAvailablePublicIps:824-IpAddressManagerImpl.fetchNewPublicIp:811-IpAddressManagerImpl.fetchNewPublicIp:803-IpAddressManagerImpl.assignPublicIpAddress:784-PublicNetworkGuru.getIp:125-PublicNetworkGuru.allocate:172-NetworkOrchestrator.allocateNic:973-NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice:823
> 2022-08-31 18:44:21,219 TRACE [c.c.u.d.T.Lock]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) SQL
> com.mysql.cj.jdbc.ClientPreparedStatement: SELECT user_ip_address.id,
> user_ip_address.account_id, user_ip_address.domain_id,
> user_ip_address.public_ip_address, user_ip_address.data_center_id,
> user_ip_address.source_nat, user_ip_address.allocated,
> user_ip_address.vlan_db_id, user_ip_address.one_to_one_nat,
> user_ip_address.vm_id, user_ip_address.state, user_ip_address.mac_address,
> user_ip_address.source_network_id, user_ip_address.network_id,
> user_ip_address.uuid, user_ip_address.physical_network_id,
> user_ip_address.is_system, user_ip_address.vpc_id,
> user_ip_address.dnat_vmip, user_ip_address.is_portable,
> user_ip_address.display, user_ip_address.rule_state,
> user_ip_address.forsystemvms, user_ip_address.removed,
> user_ip_address.created FROM user_ip_address INNER JOIN vlan ON
> user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id =
> 1 AND user_ip_address.allocated IS NULL AND user_ip_address.vlan_db_id IN
> (3,10,13,14) AND user_ip_address.forsystemvms = 1 AND
> user_ip_address.removed IS NULL AND (vlan.vlan_type = 'VirtualNetwork' )
> ORDER BY user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id ASC
> LIMIT 0, 1 FOR UPDATE took 4
> 2022-08-31 18:44:21,219 TRACE [c.c.u.d.T.Connection]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing DB connection:
> dbconn623913035
> 2022-08-31 18:44:21,220 WARN [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Unable to allocate proxy
> [Console VM instance {id: "86", name: "v-86-VM", uuid:
> "6f77acb4-c65e-49e8-ae03-1175b3974412", type="ConsoleProxy"}] on zone [1]
> due to [Insufficient address capacity].
> com.cloud.exception.InsufficientAddressCapacityException: Insufficient
> address capacityScope=interface com.cloud.dc.DataCenter; id=1
>
> following the settings of our global config which are mentioned in the
> above log:
> *system.vm.public.ip.reservation.mode.strictness* - true
> *use.system.public.ips* - true
> *enable.account.settings.for.domain* - false
>
> The challange now is to get all the things together into a context...
> From my impression changeing the value for 'use.system.public.ips' may not
> be helping, as at this point the "warning" already has been dropped. But
> would only be a "workaround".
>
> Questions from my side:
> - Looking into the web GUI for the IP-Ranges for the public network i am
> getting quite curios. The current entry for '[Root] system' is strange.
> Clicking on it i am getting a view stateing that this range shall be
> "connected" to the Account 'System', the Domain 'Root' and SystemVMs 'true'.
> While the 'System VMs = true' value can be found in the database, the vlan
> / ip - range isn't connected to a account or domain at all in the database.
> - What is the overall 'effect' of setting *system.vm.public.ip.reservation.mode.strictness?
> *From the description for me a "true" will prevent the usage of of public
> IPs from the system pool. But what is defined as "system Pool"? Looking at
> the GUI I would assume that the "account" System has a dedicated pool of
> public IPs for systemVMs (speaking that this would fail as there is no
> reference in the DB for a mapping of any Pool / Range for the account
> 'system').
>
> To summerize this seems more like a "unlucky" configuration or missing
> db.entry / bug throughout versions. Or a mix of both.
> How ever i don't really understand why there is a query to find VLANs not
> attached to a Account or an Domain - or is this the SystemPool?
>
> Thanks again for all the help!
> Chris
>
>
>
>
>
> Am Mi., 31. Aug. 2022 um 10:52 Uhr schrieb Slavka Peleva <
> slavkap@storpool.com>:
>
>> Hi Chris,
>>
>> According to the logs you shared and looking at the code, I guess CS gets
>> the VLAN IDs by domain id, not by zone id. But to be more precise in this,
>> can you enable the TRACE level in your logs and share the exact query that
>> CS executes?
>> If I'm right with this, you can enable *just for now* the setting
>> `use.system.public.ips` (To redeploy the system VMs). This should pick
>> VLANs by zone ID. I hope that the deployment won't fail this time! If the
>> system VMs are recreated, can you disable the setting
>> `use.system.public.ips` (because I don't know if this will affect something
>> else)?
>>
>> The discussion started about a problem migrating the system VMs, and
>> probably many other issues will appear.
>> Can you tell from which version you started the upgrade? (two weeks ago,
>> I upgraded one of my environments from 4.11.3.0 to the main branch
>> 4.18.0.0-SNAPSHOT without any issues)
>> Did the upgrade was successful for the management/agents?
>> Did you restart the agents on all machines? My guess is the new code
>> isn't loaded. It's strange that CS is looking for systemvm.iso when 4.17
>> doesn't know about it.
>> Can you share the content in the directory
>> `/usr/share/cloudstack-agent/lib/` from the hypervisor where the migration
>> failed?
>>
>> Best regards,
>> Slavka
>>
>>
>>
>> On Wed, Aug 31, 2022 at 2:35 AM Vash_X@gmx.de <Va...@gmx.de> wrote:
>>
>>> Hi Nicolas,
>>>
>>> > What is the value for the setting
>>> ‘system.vm.public.ip.reservation.mode.strictness’?
>>> This is set to 'true'.
>>>
>>> > Can you please check if this query gets any result?
>>> Yes the query gives me results - all availeable "public" ip addresses of
>>> the reserved range for the systemVMs
>>>
>>> # public_ip_address vlan_db_id vlan_id
>>> 1 10.129.0.10 9 1000
>>> 2 10.129.0.11 9 1000
>>> 3 10.129.0.12 9 1000
>>> 4 10.129.0.13 9 1000
>>> 5 10.129.0.14 9 1000
>>> 6 10.129.0.15 9 1000
>>> 7 10.129.0.16 9 1000
>>> 8 10.129.0.18 9 1000
>>> 9 10.129.0.19 9 1000
>>> Regarding the vlanId id=[3, 10, 13, 14]
>>> I looked up my table "vlan" and am quite shocked:
>>>
>>> the ids 3, 13, 14 have a value for "range removed", so i don't know why
>>> the
>>> console proxy is looking for these networks.
>>> the id "10" is is still active - however the ipv4 range is not "the right
>>> one". That is assigned as the public ip-range for the root-account but
>>> not
>>> for use of systemvms...
>>>
>>> Am Di., 30. Aug. 2022 um 16:43 Uhr schrieb Nicolas Vazquez <
>>> Nicolas.Vazquez@shapeblue.com>:
>>>
>>> > Hi Chris,
>>> >
>>> > What is the value for the setting
>>> > ‘system.vm.public.ip.reservation.mode.strictness’?
>>> >
>>> > Looks like the error is not being able to get an IP address for zone
>>> id=1,
>>> > vlanId id=[3, 10, 13, 14].
>>> > If I understood correctly you have defined a range for system VMs? Can
>>> you
>>> > please check if this query gets any result?
>>> >
>>> > select i.public_ip_address, i.vlan_db_id, v.vlan_id from
>>> user_ip_address i
>>> > join vlan v on (i.vlan_db_id = v.id)
>>> > where i.forsystemvms = 1 and i.allocated is null and i.state = 'Free'
>>> and
>>> > v.removed is null and i.data_center_id = 1;
>>> >
>>> > Regards,
>>> > Nicolas Vazquez
>>> >
>>> >
>>> > From: Vash_X@gmx.de <Va...@gmx.de>
>>> > Date: Tuesday, 30 August 2022 at 09:38
>>> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>, Slavka
>>> > Peleva <sl...@storpool.com>
>>> > Subject: Re: Error migrate System VMs after upgrageing to 4.17
>>> > Hi Slavka,
>>> >
>>> > that was my first thought, too. I checked beforehand and confirmed with
>>> > your provided SQL Statements:
>>> > *`select * from user_ip_address where forsystemvms = 1;`* <----
>>> returns my
>>> > defined (Public) IPs out of the correct range for SystemVMs
>>> > *`SELECT * FROM clou*d.user_ip_address where allocated IS NULL AND
>>> removed
>>> > IS NULL; <-- All IPs shown beside the one which is currently in use
>>> for the
>>> > running ssvm. Meaning from my point of view "free" for allocation for
>>> other
>>> > system VMs.
>>> >
>>> > Querys performed directly on the DB-Server.
>>> > I can't see a reason why there shouldn't be any IPs left for
>>> allocation.
>>> >
>>> > Also it is interesting why the "reasoning" for the failing deployment
>>> > changes from "no IP availeable" to "no useable volumes found".
>>> > How ever i can't see anything missing at the moment, what makes me even
>>> > more puzzeled.
>>> >
>>> > Regards
>>> > Chris
>>> >
>>> > Am Di., 30. Aug. 2022 um 13:31 Uhr schrieb Slavka Peleva
>>> > <sl...@storpool.com.invalid>:
>>> >
>>> > > Hi Chris,
>>> > >
>>> > > The deployment fails because CS cannot find a free IP address to
>>> allocate
>>> > > for the system VM. Can you check if you explicitly added a range for
>>> the
>>> > > system VMs, or you can check with this `select * from user_ip_address
>>> > where
>>> > > forsystemvms = 1;` If you don't have any ranges, you can get free IPs
>>> > > with `SELECT
>>> > > * FROM cloud.user_ip_address where allocated IS NULL AND removed IS
>>> > NULL;`
>>> > > . To get the exact query executed by CS, you need to increase the log
>>> > level
>>> > > to TRACE.
>>> > >
>>> > > Best regards,
>>> > > Slavka
>>> > >
>>> >
>>> >
>>> >
>>> > > On Mon, Aug 29, 2022 at 8:03 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
>>> > >
>>> > > > Hi Slavka,
>>> > > >
>>> > > > no, i am not using any kind of custome offerings.
>>> > > >
>>> > > > I try to provide a more detailled log this time.
>>> > > > Situation:
>>> > > > v-83-VM was the "broken" consoleproxy. So the following log starts
>>> with
>>> > > > it's expunging.
>>> > > > Afterwards the new consoleproxy shall be deployed - v-85-VM.
>>> > > >
>>> > > > This first deployment (logid:dcfe0999) fails due to Insufficent
>>> address
>>> > > > capacity - which i can't really understand or see in the database,
>>> > > checking
>>> > > > the availeable IPs for systemVMs (given that i might looking up the
>>> > wrong
>>> > > > tables....).
>>> > > > How ever CS is working as expected trying to deploy the console
>>> proxy
>>> > > > (logid:440fd101) Sync job-4288).
>>> > > > That job fails with "Unable to create deployment, no usable volumes
>>> > found
>>> > > > for the VM: 85".
>>> > > > Afterwards i have another try to start the consoleproxy (job-4289)
>>> > which
>>> > > > fails for the same reason. And thats how things go on in the log
>>> > file...
>>> > > >
>>> > > >
>>> > > > 2022-08-29 17:01:16,224 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>>> > (logid:5fe8559a)
>>> > > > Cleaning up hypervisor data structures (ex. SRs in XenServer) for
>>> > managed
>>> > > > storage. Data from VM instance {id: "83", name: "v-83-VM", uuid:
>>> > > > "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}.
>>> > > > 2022-08-29 17:01:16,225 DEBUG [o.a.c.e.o.VolumeOrchestrator]
>>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>>> > (logid:5fe8559a)
>>> > > > Cleaning storage for vm: 83
>>> > > > 2022-08-29 17:01:16,233 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>>> > (logid:5fe8559a)
>>> > > > Expunged VM instance {id: "83", name: "v-83-VM", uuid:
>>> > > > "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
>>> > > > 2022-08-29 17:01:16,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>>> > (logid:5fe8559a)
>>> > > > Complete async job-4286, jobStatus: SUCCEEDED, resultCode: 0,
>>> result:
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.api.response.SystemVmResponse/systemvm/{"id":"b1d1f9c6-eafe-4527-9335-e649c646aab0","systemvmtype":"consoleproxy","zoneid":"48e2e928-3300-43b5-8e3a-d9b358022502","zonename":"xxxxxxxxxx","dns1":"10.129.0.4","dns2":"195.10.195.195","name":"v-83-VM","templateid":"c20799e4-a990-449a-aae6-4b67374c7aff","templatename":"systemvm-kvm-4.17.0","created":"2022-08-29T10:38:26+0200","state":"Stopped","isdynamicallyscalable":"false","hasannotations":"false"}
>>> > > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>>> > (logid:5fe8559a)
>>> > > > Publish async job-4286 complete on message bus
>>> > > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>>> > (logid:5fe8559a)
>>> > > > Wake up jobs related to job-4286
>>> > > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>>> > (logid:5fe8559a)
>>> > > > Update db status for job-4286
>>> > > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>>> > (logid:5fe8559a)
>>> > > > Wake up jobs joined with job-4286 and disjoin all subjobs created
>>> from
>>> > > job-
>>> > > > 4286
>>> > > > 2022-08-29 17:01:16,260 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286) (logid:5fe8559a) Done
>>> > > executing
>>> > > >
>>> org.apache.cloudstack.api.command.admin.systemvm.DestroySystemVmCmd for
>>> > > > job-4286
>>> > > > 2022-08-29 17:01:16,260 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286) (logid:5fe8559a) Remove
>>> > > > job-4286 from job monitoring
>>> > > > 2022-08-29 17:01:17,691 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-6:ctx-6af18142) (logid:fe263a70) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > >
>>> > > > 2022-08-29 17:01:21,694 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-6:ctx-89b1442f) (logid:0efc40b2) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:01:25,698 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-5:ctx-62313d05) (logid:7073663f) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:01:25,767 DEBUG [c.c.s.StatsCollector]
>>> > > > (secstorage-1:ctx-c80e5a6e) (logid:f8e3eb60) Verifying image
>>> storage
>>> > [1].
>>> > > > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
>>> > > > 2022-08-29 17:01:25,773 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
>>> > > > (secstorage-1:ctx-c80e5a6e) (logid:f8e3eb60) Zone [1] is ready to
>>> > launch
>>> > > > secondary storage VM.
>>> > > > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Zone 1 is ready to
>>> > launch
>>> > > > console proxy
>>> > > > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Expand console proxy
>>> > > standby
>>> > > > capacity for zone xxxxxxxxxx
>>> > > > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating console
>>> proxy
>>> > > > standby capacity for zone [1].
>>> > > > 2022-08-29 17:01:26,032 INFO [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) No stopped console
>>> proxy
>>> > > is
>>> > > > available, need to allocate a new console proxy
>>> > > > 2022-08-29 17:01:26,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Assign console proxy
>>> > from
>>> > > a
>>> > > > newly started instance for request from data center : 1
>>> > > > 2022-08-29 17:01:26,056 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing
>>> network
>>> > > > configuration for offering [Network Offering
>>> > > > [1-Public-System-Public-Network]: Network {"id": 200, "name":
>>> "null",
>>> > > > "uuid": "1b0bf965-a391-478d-833d-3e37e4a15eb8",
>>> "networkofferingid": 1}
>>> > > > 2022-08-29 17:01:26,056 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
>>> > > > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account
>>> {"id": 1,
>>> > > > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
>>> > > > 2022-08-29 17:01:26,064 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing
>>> network
>>> > > > configuration for offering [Network Offering
>>> > > > [3-Control-System-Control-Network]: Network {"id": 202, "name":
>>> "null",
>>> > > > "uuid": "2af42518-fbf4-4a35-9d6b-370e6ad315c7",
>>> "networkofferingid": 3}
>>> > > > 2022-08-29 17:01:26,064 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
>>> > > > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account
>>> {"id": 1,
>>> > > > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
>>> > > > 2022-08-29 17:01:26,069 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing
>>> network
>>> > > > configuration for offering [Network Offering
>>> > > > [2-Management-System-Management-Network]: Network {"id": 201,
>>> "name":
>>> > > > "null", "uuid": "53711d87-a3c6-4f33-bfce-c30e0de72760",
>>> > > > "networkofferingid": 2}
>>> > > > 2022-08-29 17:01:26,069 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
>>> > > > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account
>>> {"id": 1,
>>> > > > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
>>> > > > 2022-08-29 17:01:26,077 INFO [c.c.v.VirtualMachineManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) allocating virtual
>>> > machine
>>> > > > from template:c20799e4-a990-449a-aae6-4b67374c7aff with
>>> > hostname:v-85-VM
>>> > > > and 3 networks
>>> > > > 2022-08-29 17:01:26,079 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating entries
>>> for
>>> > VM:
>>> > > > VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
>>> > > > 2022-08-29 17:01:26,082 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating nics for
>>> VM
>>> > > > instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
>>> > > > 2022-08-29 17:01:26,083 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating nic for
>>> vm VM
>>> > > > instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} in
>>> network
>>> > > > Network {"id": 200, "name": "null", "uuid":
>>> > > > "1b0bf965-a391-478d-833d-3e37e4a15eb8", "networkofferingid": 1}
>>> with
>>> > > > requested profile NicProfile
>>> > > >
>>> > >
>>> >
>>> {"broadcastUri":null,"iPv4Address":null,"id":0,"reservationId":null,"vmId":0}
>>> > > > 2022-08-29 17:01:26,089 WARN [c.c.n.IpAddressManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to get ip
>>> address
>>> > > in
>>> > > > zone id=1, vlanId id=[3, 10, 13, 14]
>>> > > > 2022-08-29 17:01:26,090 DEBUG [c.c.u.d.T.Transaction]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Rolling back the
>>> > > > transaction: Time = 8 Name = consoleproxy-1; called by
>>> > > >
>>> > >
>>> >
>>> -TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-Transaction.execute:38-IpAddressManagerImpl.listAvailablePublicIps:824-IpAddressManagerImpl.fetchNewPublicIp:811-IpAddressManagerImpl.fetchNewPublicIp:803-IpAddressManagerImpl.assignPublicIpAddress:784-PublicNetworkGuru.getIp:125-PublicNetworkGuru.allocate:172-NetworkOrchestrator.allocateNic:973-NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice:823
>>> > > > 2022-08-29 17:01:26,090 WARN [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to allocate
>>> proxy
>>> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
>>> zone
>>> > [1]
>>> > > > due to [Insufficient address capacity].
>>> > > > com.cloud.exception.InsufficientAddressCapacityException:
>>> Insufficient
>>> > > > address capacityScope=interface com.cloud.dc.DataCenter; id=1
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
>>> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.IpAddressManagerImpl.listAvailablePublicIps(IpAddressManagerImpl.java:824)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl.java:811)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl.java:803)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.IpAddressManagerImpl.assignPublicIpAddress(IpAddressManagerImpl.java:784)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.guru.PublicNetworkGuru.getIp(PublicNetworkGuru.java:125)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.guru.PublicNetworkGuru.allocate(PublicNetworkGuru.java:172)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocateNic(NetworkOrchestrator.java:973)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice(NetworkOrchestrator.java:823)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$3.doInTransactionWithoutResult(NetworkOrchestrator.java:797)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
>>> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocate(NetworkOrchestrator.java:778)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl$1.doInTransactionWithoutResult(VirtualMachineManagerImpl.java:478)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
>>> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:469)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:528)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:700)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
>>> > > > at
>>> > >
>>> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
>>> > > > at
>>> > > >
>>> >
>>> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>>> > > > 2022-08-29 17:01:26,091 WARN [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to allocate
>>> > console
>>> > > > proxy standby capacity for zone [1] due to [Unable to allocate
>>> proxy
>>> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
>>> zone
>>> > [1]
>>> > > > due to [Insufficient address capacity].].
>>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to allocate
>>> > proxy
>>> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
>>> zone
>>> > [1]
>>> > > > due to [Insufficient address capacity].
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:704)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
>>> > > > at
>>> > >
>>> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
>>> > > > at
>>> > > >
>>> >
>>> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>>> > > > Caused by:
>>> com.cloud.exception.InsufficientAddressCapacityException:
>>> > > > Insufficient address capacityScope=interface
>>> com.cloud.dc.DataCenter;
>>> > > id=1
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
>>> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
>>> > > > ... 38 more
>>> > > > 2022-08-29 17:01:26,091 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) received console
>>> proxy
>>> > > alert
>>> > > > 2022-08-29 17:01:26,092 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Console proxy
>>> creation
>>> > > > failure, zone: xxxxxxxxxx
>>> > > > 2022-08-29 17:01:26,093 WARN [c.c.a.AlertManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) alertType=[10]
>>> > > > dataCenterId=[1] podId=[null] clusterId=[null] message=[Console
>>> proxy
>>> > > > creation failure. zone: xxxxxxxxxx, error details: Unable to
>>> allocate
>>> > > proxy
>>> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
>>> zone
>>> > [1]
>>> > > > due to [Insufficient address capacity].].
>>> > > > 2022-08-29 17:01:26,096 WARN [c.c.a.AlertManagerImpl]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) No recipients set in
>>> > > global
>>> > > > setting 'alert.email.addresses', skipping sending alert with
>>> subject
>>> > > > [Console proxy creation failure. zone: xxxxxxxxxx, error details:
>>> > Unable
>>> > > to
>>> > > > allocate proxy [Console VM instance {id: "85", name: "v-85-VM",
>>> uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
>>> zone
>>> > [1]
>>> > > > due to [Insufficient address capacity].] and content [Console proxy
>>> > > > creation failure (zone xxxxxxxxxx)].
>>> > > > 2022-08-29 17:01:26,096 WARN [c.c.v.SystemVmLoadScanner]
>>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unexpected exception
>>> > > Unable
>>> > > > to allocate proxy [Console VM instance {id: "85", name: "v-85-VM",
>>> > uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
>>> zone
>>> > [1]
>>> > > > due to [Insufficient address capacity].
>>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to allocate
>>> > proxy
>>> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
>>> zone
>>> > [1]
>>> > > > due to [Insufficient address capacity].
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:704)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
>>> > > > at
>>> > >
>>> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
>>> > > > at
>>> > > >
>>> >
>>> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>>> > > > Caused by:
>>> com.cloud.exception.InsufficientAddressCapacityException:
>>> > > > Insufficient address capacityScope=interface
>>> com.cloud.dc.DataCenter;
>>> > > id=1
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
>>> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
>>> > > > ... 38 more
>>> > > > 2022-08-29 17:01:29,700 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-5:ctx-994465bd) (logid:1e2822fd) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:01:30,444 DEBUG
>>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
>>> > > > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 2
>>> routers
>>> > to
>>> > > > update status.
>>> > > > 2022-08-29 17:01:30,449 DEBUG
>>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
>>> > > > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 1
>>> VPC's to
>>> > > > update Redundant State.
>>> > > > 2022-08-29 17:01:30,451 DEBUG
>>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
>>> > > > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 0
>>> networks
>>> > to
>>> > > > update RvR status.
>>> > > > 2022-08-29 17:01:30,459 DEBUG [c.c.a.t.Request]
>>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
>>> > > > 6-4833769775052102463: Sending { Cmd , MgmtId: 90520740759984,
>>> via:
>>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
>>> > > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
>>> > > router.name
>>> > >
>>> >
>>> ":"r-79-VM","router.ip":"169.254.144.198"},"wait":"30","bypassHostMaintenance":"false"}}]
>>> > > > }
>>> > > > 2022-08-29 17:01:30,643 DEBUG [c.c.a.t.Request]
>>> > > > (AgentManager-Handler-12:null) (logid:) Seq 6-4833769775052102463:
>>> > > > Processing: { Ans: , MgmtId: 90520740759984, via: 6, Ver: v1,
>>> Flags:
>>> > 10,
>>> > > >
>>> > >
>>> >
>>> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"PRIMARY","result":"true","details":"Status:
>>> > > > PRIMARY
>>> > > > ","wait":"0","bypassHostMaintenance":"false"}}] }
>>> > > > 2022-08-29 17:01:30,643 DEBUG [c.c.a.t.Request]
>>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
>>> > > > 6-4833769775052102463: Received: { Ans: , MgmtId: 90520740759984,
>>> via:
>>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>>> CheckRouterAnswer } }
>>> > > > 2022-08-29 17:01:30,643 DEBUG [c.c.a.m.AgentManagerImpl]
>>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08)
>>> Details
>>> > > from
>>> > > > executing class com.cloud.agent.api.CheckRouterCommand: Status:
>>> PRIMARY
>>> > > >
>>> > > > 2022-08-29 17:01:30,655 DEBUG [c.c.a.t.Request]
>>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
>>> > > > 1-5410793477308947401: Sending { Cmd , MgmtId: 90520740759984,
>>> via:
>>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
>>> > > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
>>> > > router.name
>>> > >
>>> >
>>> ":"r-80-VM","router.ip":"169.254.217.200"},"wait":"30","bypassHostMaintenance":"false"}}]
>>> > > > }
>>> > > > 2022-08-29 17:01:30,796 DEBUG [c.c.a.t.Request]
>>> > > > (AgentManager-Handler-7:null) (logid:) Seq 1-5410793477308947401:
>>> > > > Processing: { Ans: , MgmtId: 90520740759984, via: 1, Ver: v1,
>>> Flags:
>>> > 10,
>>> > > >
>>> > >
>>> >
>>> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"BACKUP","result":"true","details":"Status:
>>> > > > BACKUP
>>> > > > ","wait":"0","bypassHostMaintenance":"false"}}] }
>>> > > > 2022-08-29 17:01:30,796 DEBUG [c.c.a.t.Request]
>>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
>>> > > > 1-5410793477308947401: Received: { Ans: , MgmtId: 90520740759984,
>>> via:
>>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>>> CheckRouterAnswer } }
>>> > > > 2022-08-29 17:01:30,796 DEBUG [c.c.a.m.AgentManagerImpl]
>>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08)
>>> Details
>>> > > from
>>> > > > executing class com.cloud.agent.api.CheckRouterCommand: Status:
>>> BACKUP
>>> > > >
>>> > > > 2022-08-29 17:01:33,703 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-5:ctx-70bca202) (logid:b332dbe0) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:01:34,083 DEBUG [c.c.s.StatsCollector]
>>> > > > (StatsCollector-6:ctx-213e2ecc) (logid:aa0584c9) AutoScaling
>>> Monitor is
>>> > > > running...
>>> > > > 2022-08-29 17:01:37,707 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-6:ctx-57493517) (logid:5093e75f) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:01:38,640 DEBUG [c.c.s.StatsCollector]
>>> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c)
>>> HostStatsCollector is
>>> > > > running...
>>> > > > 2022-08-29 17:01:38,655 DEBUG [c.c.a.t.Request]
>>> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
>>> > > 1-5410793477308947402:
>>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>>> GetHostStatsAnswer }
>>> > }
>>> > > > 2022-08-29 17:01:38,707 DEBUG [c.c.a.t.Request]
>>> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
>>> > > 5-8959067033723473395:
>>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>>> > > > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>>> GetHostStatsAnswer }
>>> > }
>>> > > > 2022-08-29 17:01:38,717 DEBUG [c.c.a.t.Request]
>>> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
>>> > > 6-4833769775052102464:
>>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>>> GetHostStatsAnswer }
>>> > }
>>> > > > 2022-08-29 17:01:38,725 DEBUG [c.c.a.t.Request]
>>> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
>>> > > 7-295548725546192472:
>>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>>> > > > 7(srv-5-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>>> GetHostStatsAnswer }
>>> > }
>>> > > > 2022-08-29 17:01:41,709 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-5:ctx-a65ae462) (logid:bc2b7320) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:01:45,712 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-2:ctx-05491ace) (logid:b2232c4d) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:01:47,019 DEBUG [c.c.a.m.AgentManagerImpl]
>>> > > > (AgentManager-Handler-14:null) (logid:) Ping from Routing host
>>> > > > 7(srv-5-comp.aaaaaaaa.bbb)
>>> > > >
>>> > > > 2022-08-29 17:01:49,715 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-2:ctx-7943581c) (logid:504c98ff) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:01:51,881 DEBUG [c.c.a.m.AgentManagerImpl]
>>> > > > (AgentManager-Handler-13:null) (logid:) Ping from Routing host
>>> > > > 1(srv-2-comp.aaaaaaaa.bbb)
>>> > > >
>>> > > > 2022-08-29 17:01:53,718 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-4:ctx-b11cfa22) (logid:3163613f) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:01:53,955 DEBUG [c.c.a.t.Request]
>>> > > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
>>> > > 1-5410793477308947403:
>>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer
>>> } }
>>> > > > 2022-08-29 17:01:54,025 DEBUG [c.c.a.t.Request]
>>> > > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
>>> > > 5-8959067033723473396:
>>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>>> > > > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer
>>> } }
>>> > > > 2022-08-29 17:01:54,096 DEBUG [c.c.a.t.Request]
>>> > > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
>>> > > 6-4833769775052102465:
>>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer
>>> } }
>>> > > > 2022-08-29 17:01:54,217 DEBUG [c.c.a.t.Request]
>>> > > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
>>> > > 7-295548725546192473:
>>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>>> > > > 7(srv-5-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer
>>> } }
>>> > > > 2022-08-29 17:01:55,765 DEBUG [c.c.s.StatsCollector]
>>> > > > (secstorage-1:ctx-7d9995d7) (logid:e388a8b8) Verifying image
>>> storage
>>> > [1].
>>> > > > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
>>> > > > 2022-08-29 17:01:55,771 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
>>> > > > (secstorage-1:ctx-7d9995d7) (logid:e388a8b8) Zone [1] is ready to
>>> > launch
>>> > > > secondary storage VM.
>>> > > > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Zone 1 is ready to
>>> > launch
>>> > > > console proxy
>>> > > > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Expand console proxy
>>> > > standby
>>> > > > capacity for zone xxxxxxxxxx
>>> > > > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Allocating console
>>> proxy
>>> > > > standby capacity for zone [1].
>>> > > > 2022-08-29 17:01:56,034 INFO [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Found a stopped
>>> console
>>> > > > proxy, starting it. Vm id : 85
>>> > > > 2022-08-29 17:01:56,035 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) start parameter
>>> value of
>>> > > > enterHardwareSetup == <very null> during processing of queued job
>>> > > > 2022-08-29 17:01:56,044 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Sync job-4288
>>> execution
>>> > on
>>> > > > object VmWorkJobQueue.85
>>> > > > 2022-08-29 17:01:56,173 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (AsyncJobMgr-Heartbeat-1:ctx-e75826b1) (logid:e80fe659) Execute
>>> > > sync-queue
>>> > > > item: SyncQueueItemVO {id:2105, queueId: 2105, contentType:
>>> AsyncJob,
>>> > > > contentId: 4288, lastProcessMsid: 90520740759984,
>>> lastprocessNumber: 1,
>>> > > > lastProcessTime: Mon Aug 29 17:01:56 CEST 2022, created: Mon Aug 29
>>> > > > 17:01:56 CEST 2022}
>>> > > > 2022-08-29 17:01:56,174 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (AsyncJobMgr-Heartbeat-1:ctx-e75826b1) (logid:e80fe659) Schedule
>>> queued
>>> > > > job-4288
>>> > > > 2022-08-29 17:01:56,179 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
>>> (logid:7e671dd0)
>>> > > Add
>>> > > > job-4288 into job monitoring
>>> > > > 2022-08-29 17:01:56,186 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
>>> (logid:ba10fec5)
>>> > > > Executing AsyncJobVO: {id:4288, userId: 1, accountId: 1,
>>> instanceType:
>>> > > > null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
>>> > > >
>>> > >
>>> >
>>> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAVXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
>>> > > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
>>> 0,
>>> > > > result: null, initMsid: 90520740759984, completeMsid: null,
>>> > lastUpdated:
>>> > > > null, lastPolled: null, created: Mon Aug 29 17:01:56 CEST 2022,
>>> > removed:
>>> > > > null}
>>> > > > 2022-08-29 17:01:56,186 DEBUG [c.c.v.VmWorkJobDispatcher]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
>>> (logid:ba10fec5)
>>> > > Run
>>> > > > VM work job: com.cloud.vm.VmWorkStart for VM 85, job origin: 3445
>>> > > > 2022-08-29 17:01:56,187 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Execute VM work job:
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
>>> > > > 2022-08-29 17:01:56,189 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) orchestrating VM start for 'v-85-VM'
>>> > > > com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
>>> > > > 2022-08-29 17:01:56,196 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
>>> > > > transited from [Stopped] to [Starting] with event [StartRequested].
>>> > VM's
>>> > > > original host: null, new host: null, host before state transition:
>>> null
>>> > > > 2022-08-29 17:01:56,196 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Successfully transitioned to start state for VM
>>> > instance
>>> > > > {id: "85", name: "v-85-VM", uuid:
>>> > "110d8620-c029-4ca9-9084-210099725a80",
>>> > > > type="ConsoleProxy"} reservation id =
>>> > > 0ab68202-d8be-469c-9b2a-87a55c2dc489
>>> > > > 2022-08-29 17:01:56,200 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts:
>>> null
>>> > > > 2022-08-29 17:01:56,202 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) VM start attempt #1
>>> > > > 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
>>> > > > 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Trying to allocate a host and storage pools from
>>> dc:1,
>>> > > > pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
>>> > > > 1073741824
>>> > > > 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
>>> > > > 2022-08-29 17:01:56,212 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
>>> > > > 2022-08-29 17:01:56,214 DEBUG [c.c.d.FirstFitPlanner]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Searching all possible resources under this Zone:
>>> 1
>>> > > > 2022-08-29 17:01:56,215 DEBUG [c.c.d.FirstFitPlanner]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Listing clusters in order of aggregate capacity,
>>> that
>>> > > have
>>> > > > (at least one host with) enough CPU and RAM capacity under this
>>> Zone: 1
>>> > > > 2022-08-29 17:01:56,218 DEBUG [c.c.d.FirstFitPlanner]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Removing from the clusterId list these clusters
>>> from
>>> > > avoid
>>> > > > set: []
>>> > > > 2022-08-29 17:01:56,224 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
>>> > > > 2022-08-29 17:01:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested
>>> with
>>> > > > Custom[UEFI] Boot Type false
>>> > > > 2022-08-29 17:01:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in
>>> dc: 1
>>> > > > pod:1 cluster:1
>>> > > > 2022-08-29 17:01:56,226 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4
>>> > hosts
>>> > > to
>>> > > > check for allocation: [Host {"id": "5", "name":
>>> > > "srv-4-comp.aaaaaaaa.bbb",
>>> > > > "uuid": "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"},
>>> Host
>>> > > > {"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
>>> {"id":
>>> > > "6",
>>> > > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
>>> {"id":
>>> > > "1",
>>> > > > "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
>>> > > > 2022-08-29 17:01:56,232 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for
>>> allocation
>>> > > > after prioritization: [Host {"id": "5", "name":
>>> > > "srv-4-comp.aaaaaaaa.bbb",
>>> > > > "uuid": "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"},
>>> Host
>>> > > > {"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
>>> {"id":
>>> > > "6",
>>> > > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
>>> {"id":
>>> > > "1",
>>> > > > "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
>>> > > > 2022-08-29 17:01:56,232 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for
>>> speed=500Mhz,
>>> > > > Ram=1024 MB
>>> > > > 2022-08-29 17:01:56,232 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
>>> > > > srv-4-comp.aaaaaaaa.bbb, uuid:
>>> 134c625d-15ab-461a-a38d-36fb0b979b5d} is
>>> > > KVM
>>> > > > hypervisor type, no max guest limit check needed
>>> > > > 2022-08-29 17:01:56,234 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu
>>> capability
>>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> speed:
>>> > 500
>>> > > > 2022-08-29 17:01:56,234 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has
>>> > enough
>>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > 1073741824 ,
>>> > > > cpuOverprovisioningFactor: 1.0
>>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> CPU:
>>> > > 127200
>>> > > > and CPU after applying overprovisioning: 127200
>>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 ,
>>> Requested
>>> > > > CPU: 500
>>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
>>> > > > 251009040384 , Requested RAM: (1.00 GB) 1073741824
>>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>>> RAM
>>> > > > available
>>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> from
>>> > > host:
>>> > > > 5, used: 11100, reserved: 0, actual total: 127200, total with
>>> > > > overprovisioning: 127200; requested
>>> cpu:500,alloc_from_last_host?:false
>>> > > > ,considerReservedCapacity?: true
>>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> from
>>> > > host:
>>> > > > 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total:
>>> (250.27
>>> > > GB)
>>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > alloc_from_last_host?:
>>> > > > false , considerReservedCapacity?: true
>>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>>> > adding
>>> > > to
>>> > > > list: 5
>>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
>>> > > > srv-5-comp.aaaaaaaa.bbb, uuid:
>>> 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is
>>> > > KVM
>>> > > > hypervisor type, no max guest limit check needed
>>> > > > 2022-08-29 17:01:56,238 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu
>>> capability
>>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> speed:
>>> > 500
>>> > > > 2022-08-29 17:01:56,238 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has
>>> > enough
>>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > 1073741824 ,
>>> > > > cpuOverprovisioningFactor: 1.0
>>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> CPU:
>>> > > 127200
>>> > > > and CPU after applying overprovisioning: 127200
>>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 ,
>>> Requested
>>> > > CPU:
>>> > > > 500
>>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
>>> > > > 189373743104 , Requested RAM: (1.00 GB) 1073741824
>>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>>> RAM
>>> > > > available
>>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> from
>>> > > host:
>>> > > > 7, used: 90100, reserved: 0, actual total: 127200, total with
>>> > > > overprovisioning: 127200; requested
>>> cpu:500,alloc_from_last_host?:false
>>> > > > ,considerReservedCapacity?: true
>>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> from
>>> > > host:
>>> > > > 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total:
>>> (250.27
>>> > > GB)
>>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > alloc_from_last_host?:
>>> > > > false , considerReservedCapacity?: true
>>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>>> > adding
>>> > > to
>>> > > > list: 7
>>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
>>> > > > srv-3-comp.aaaaaaaa.bbb, uuid:
>>> 6858713b-7e17-46d8-854d-24a13f9c7001} is
>>> > > KVM
>>> > > > hypervisor type, no max guest limit check needed
>>> > > > 2022-08-29 17:01:56,242 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu
>>> capability
>>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> speed:
>>> > 500
>>> > > > 2022-08-29 17:01:56,242 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has
>>> > enough
>>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > 1073741824 ,
>>> > > > cpuOverprovisioningFactor: 1.0
>>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> CPU:
>>> > > 127200
>>> > > > and CPU after applying overprovisioning: 127200
>>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 ,
>>> Requested
>>> > > CPU:
>>> > > > 500
>>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
>>> > > > 204032835584 , Requested RAM: (1.00 GB) 1073741824
>>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>>> RAM
>>> > > > available
>>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> from
>>> > > host:
>>> > > > 6, used: 42900, reserved: 0, actual total: 127200, total with
>>> > > > overprovisioning: 127200; requested
>>> cpu:500,alloc_from_last_host?:false
>>> > > > ,considerReservedCapacity?: true
>>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> from
>>> > > host:
>>> > > > 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total:
>>> (250.27
>>> > > GB)
>>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > alloc_from_last_host?:
>>> > > > false , considerReservedCapacity?: true
>>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>>> > adding
>>> > > to
>>> > > > list: 6
>>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
>>> > > > srv-2-comp.aaaaaaaa.bbb, uuid:
>>> 883f7f62-55f2-42f4-8242-16b6847c4204} is
>>> > > KVM
>>> > > > hypervisor type, no max guest limit check needed
>>> > > > 2022-08-29 17:01:56,246 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu
>>> capability
>>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> speed:
>>> > 500
>>> > > > 2022-08-29 17:01:56,246 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has
>>> > enough
>>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > 1073741824 ,
>>> > > > cpuOverprovisioningFactor: 1.0
>>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> CPU:
>>> > > 127200
>>> > > > and CPU after applying overprovisioning: 127200
>>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 ,
>>> Requested
>>> > > CPU:
>>> > > > 500
>>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
>>> > > > 229802639360 , Requested RAM: (1.00 GB) 1073741824
>>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>>> RAM
>>> > > > available
>>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> from
>>> > > host:
>>> > > > 1, used: 48200, reserved: 0, actual total: 127200, total with
>>> > > > overprovisioning: 127200; requested
>>> cpu:500,alloc_from_last_host?:false
>>> > > > ,considerReservedCapacity?: true
>>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> from
>>> > > host:
>>> > > > 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total:
>>> (250.27
>>> > > GB)
>>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > alloc_from_last_host?:
>>> > > > false , considerReservedCapacity?: true
>>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>>> > adding
>>> > > to
>>> > > > list: 1
>>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator
>>> returning 4
>>> > > > suitable hosts
>>> > > > 2022-08-29 17:01:56,258 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
>>> > > > transited from [Starting] to [Stopped] with event
>>> [OperationFailed].
>>> > VM's
>>> > > > original host: null, new host: null, host before state transition:
>>> null
>>> > > > 2022-08-29 17:01:56,258 WARN [c.c.v.VirtualMachineManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Unable to orchestrate start VM instance {id: "85",
>>> > name:
>>> > > > "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
>>> > > > type="ConsoleProxy"} due to [Unable to create deployment, no usable
>>> > > volumes
>>> > > > found for the VM: 85].
>>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to create
>>> > > > deployment, no usable volumes found for the VM: 85
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
>>> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>>> > Source)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>>> > > > at
>>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> > > > at
>>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>>> > > > 2022-08-29 17:01:56,259 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Done executing VM work job:
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
>>> > > > 2022-08-29 17:01:56,260 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Complete async job-4288, jobStatus: FAILED,
>>> > resultCode:
>>> > > 0,
>>> > > > result:
>>> > > >
>>> > >
>>> >
>>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODUiLCBuYW1lOiAidi04NS1WTSIsIHV1aWQ6ICIxMTBkODYyMC1jMDI5LTRjYTktOTA4NC0yMTAwOTk3MjVhODAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODVdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
>>> > > > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Publish async job-4288 complete on message bus
>>> > > > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Wake up jobs related to job-4288
>>> > > > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Update db status for job-4288
>>> > > > 2022-08-29 17:01:56,262 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>>> > > > (logid:ba10fec5) Wake up jobs joined with job-4288 and disjoin all
>>> > > subjobs
>>> > > > created from job- 4288
>>> > > > 2022-08-29 17:01:56,270 DEBUG [c.c.v.VmWorkJobDispatcher]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
>>> (logid:ba10fec5)
>>> > > > Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 85,
>>> job
>>> > > > origin: 3445
>>> > > > 2022-08-29 17:01:56,270 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
>>> (logid:ba10fec5)
>>> > > > Done executing com.cloud.vm.VmWorkStart for job-4288
>>> > > > 2022-08-29 17:01:56,272 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
>>> (logid:ba10fec5)
>>> > > > Remove job-4288 from job monitoring
>>> > > > 2022-08-29 17:01:56,277 WARN [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Unable to allocate
>>> > console
>>> > > > proxy standby capacity for zone [1] due to [Unable to orchestrate
>>> start
>>> > > VM
>>> > > > instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>>> > > [Unable
>>> > > > to create deployment, no usable volumes found for the VM: 85].].
>>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to
>>> orchestrate
>>> > > > start VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>>> > > [Unable
>>> > > > to create deployment, no usable volumes found for the VM: 85].
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
>>> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>>> > Source)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>>> > > > at
>>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> > > > at
>>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>>> > > > 2022-08-29 17:01:56,277 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) received console
>>> proxy
>>> > > alert
>>> > > > 2022-08-29 17:01:56,278 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Console proxy
>>> creation
>>> > > > failure, zone: xxxxxxxxxx
>>> > > > 2022-08-29 17:01:56,279 WARN [c.c.a.AlertManagerImpl]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) alertType=[10]
>>> > > > dataCenterId=[1] podId=[null] clusterId=[null] message=[Console
>>> proxy
>>> > > > creation failure. zone: xxxxxxxxxx, error details: Unable to
>>> > orchestrate
>>> > > > start VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>>> > > [Unable
>>> > > > to create deployment, no usable volumes found for the VM: 85].].
>>> > > > 2022-08-29 17:01:56,284 WARN [c.c.a.AlertManagerImpl]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) No recipients set in
>>> > > global
>>> > > > setting 'alert.email.addresses', skipping sending alert with
>>> subject
>>> > > > [Console proxy creation failure. zone: xxxxxxxxxx, error details:
>>> > Unable
>>> > > to
>>> > > > orchestrate start VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>>> > > [Unable
>>> > > > to create deployment, no usable volumes found for the VM: 85].] and
>>> > > content
>>> > > > [Console proxy creation failure (zone xxxxxxxxxx)].
>>> > > > 2022-08-29 17:01:56,284 WARN [c.c.v.SystemVmLoadScanner]
>>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Unexpected exception
>>> > > Unable
>>> > > > to orchestrate start VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>>> > > [Unable
>>> > > > to create deployment, no usable volumes found for the VM: 85].
>>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to
>>> orchestrate
>>> > > > start VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>>> > > [Unable
>>> > > > to create deployment, no usable volumes found for the VM: 85].
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
>>> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>>> > Source)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>>> > > > at
>>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> > > > at
>>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>>> > > > 2022-08-29 17:01:57,721 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-4:ctx-9528226f) (logid:4512ff9b) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:01:59,504 DEBUG [c.c.s.StatsCollector]
>>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) StorageCollector
>>> is
>>> > > > running...
>>> > > > 2022-08-29 17:01:59,518 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
>>> > > getCommandHostDelegation:
>>> > > > class com.cloud.agent.api.GetStorageStatsCommand
>>> > > > 2022-08-29 17:01:59,518 DEBUG [c.c.h.XenServerGuru]
>>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning
>>> the
>>> > > > default host to execute commands because the command is not of Copy
>>> > type.
>>> > > > 2022-08-29 17:01:59,571 DEBUG [c.c.a.t.Request]
>>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
>>> > > > 11-7523826127475836583: Received: { Ans: , MgmtId: 90520740759984,
>>> > via:
>>> > > > 11(s-32-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
>>> > > > 2022-08-29 17:01:59,574 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
>>> > > getCommandHostDelegation:
>>> > > > class com.cloud.agent.api.GetStorageStatsCommand
>>> > > > 2022-08-29 17:01:59,574 DEBUG [c.c.h.XenServerGuru]
>>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning
>>> the
>>> > > > default host to execute commands because the command is not of Copy
>>> > type.
>>> > > > 2022-08-29 17:01:59,615 DEBUG [c.c.a.t.Request]
>>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
>>> > > 5-8959067033723473397:
>>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>>> > > > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>>> GetStorageStatsAnswer
>>> > > } }
>>> > > > 2022-08-29 17:01:59,624 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
>>> > > getCommandHostDelegation:
>>> > > > class com.cloud.agent.api.GetStorageStatsCommand
>>> > > > 2022-08-29 17:01:59,624 DEBUG [c.c.h.XenServerGuru]
>>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning
>>> the
>>> > > > default host to execute commands because the command is not of Copy
>>> > type.
>>> > > > 2022-08-29 17:01:59,653 DEBUG [c.c.a.t.Request]
>>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
>>> > > 6-4833769775052102466:
>>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>>> GetStorageStatsAnswer
>>> > > } }
>>> > > > 2022-08-29 17:01:59,979 DEBUG [c.c.a.m.AgentManagerImpl]
>>> > > > (AgentManager-Handler-12:null) (logid:) Ping from Routing host
>>> > > > 5(srv-4-comp.aaaaaaaa.bbb)
>>> > > >
>>> > > > 2022-08-29 17:02:00,269 DEBUG [c.c.s.StatsCollector]
>>> > > > (StatsCollector-5:ctx-d2b6ffd3) (logid:95201678) DbCollector is
>>> > > running...
>>> > > > 2022-08-29 17:02:00,443 DEBUG
>>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
>>> > > > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 2
>>> routers
>>> > to
>>> > > > update status.
>>> > > > 2022-08-29 17:02:00,449 DEBUG
>>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
>>> > > > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 1
>>> VPC's to
>>> > > > update Redundant State.
>>> > > > 2022-08-29 17:02:00,450 DEBUG
>>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
>>> > > > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 0
>>> networks
>>> > to
>>> > > > update RvR status.
>>> > > > 2022-08-29 17:02:00,457 DEBUG [c.c.a.t.Request]
>>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
>>> > > > 6-4833769775052102467: Sending { Cmd , MgmtId: 90520740759984,
>>> via:
>>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
>>> > > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
>>> > > router.name
>>> > >
>>> >
>>> ":"r-79-VM","router.ip":"169.254.144.198"},"wait":"30","bypassHostMaintenance":"false"}}]
>>> > > > }
>>> > > > 2022-08-29 17:02:00,599 DEBUG [c.c.a.t.Request]
>>> > > > (AgentManager-Handler-7:null) (logid:) Seq 6-4833769775052102467:
>>> > > > Processing: { Ans: , MgmtId: 90520740759984, via: 6, Ver: v1,
>>> Flags:
>>> > 10,
>>> > > >
>>> > >
>>> >
>>> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"PRIMARY","result":"true","details":"Status:
>>> > > > PRIMARY
>>> > > > ","wait":"0","bypassHostMaintenance":"false"}}] }
>>> > > > 2022-08-29 17:02:00,599 DEBUG [c.c.a.t.Request]
>>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
>>> > > > 6-4833769775052102467: Received: { Ans: , MgmtId: 90520740759984,
>>> via:
>>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>>> CheckRouterAnswer } }
>>> > > > 2022-08-29 17:02:00,599 DEBUG [c.c.a.m.AgentManagerImpl]
>>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f)
>>> Details
>>> > > > from executing class com.cloud.agent.api.CheckRouterCommand:
>>> Status:
>>> > > PRIMARY
>>> > > >
>>> > > > 2022-08-29 17:02:00,612 DEBUG [c.c.a.t.Request]
>>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
>>> > > > 1-5410793477308947404: Sending { Cmd , MgmtId: 90520740759984,
>>> via:
>>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
>>> > > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
>>> > > router.name
>>> > >
>>> >
>>> ":"r-80-VM","router.ip":"169.254.217.200"},"wait":"30","bypassHostMaintenance":"false"}}]
>>> > > > }
>>> > > > 2022-08-29 17:02:00,748 DEBUG [c.c.a.m.AgentManagerImpl]
>>> > > > (AgentManager-Handler-4:null) (logid:) Ping from 11(s-32-VM)
>>> > > > 2022-08-29 17:02:00,750 DEBUG [c.c.a.t.Request]
>>> > > > (AgentManager-Handler-1:null) (logid:) Seq 1-5410793477308947404:
>>> > > > Processing: { Ans: , MgmtId: 90520740759984, via: 1, Ver: v1,
>>> Flags:
>>> > 10,
>>> > > >
>>> > >
>>> >
>>> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"BACKUP","result":"true","details":"Status:
>>> > > > BACKUP
>>> > > > ","wait":"0","bypassHostMaintenance":"false"}}] }
>>> > > > 2022-08-29 17:02:00,750 DEBUG [c.c.a.t.Request]
>>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
>>> > > > 1-5410793477308947404: Received: { Ans: , MgmtId: 90520740759984,
>>> via:
>>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>>> CheckRouterAnswer } }
>>> > > > 2022-08-29 17:02:00,751 DEBUG [c.c.a.m.AgentManagerImpl]
>>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f)
>>> Details
>>> > > > from executing class com.cloud.agent.api.CheckRouterCommand:
>>> Status:
>>> > > BACKUP
>>> > > >
>>> > > > 2022-08-29 17:02:01,362 DEBUG [c.c.s.StatsCollector]
>>> > > > (Cluster-Worker-839:ctx-e097a8ab) (logid:3deacf65) StatusUpdate
>>> from
>>> > > > 90520740759984, json:
>>> > > >
>>> > >
>>> >
>>> {"managementServerHostId":1,"managementServerHostUuid":"5615ca5f-0716-4da1-93ec-31f79fd19bbe","collectionTime":"Aug
>>> > > > 29, 2022, 5:02:01
>>> > > >
>>> > >
>>> >
>>> PM","sessions":1,"cpuUtilization":0.0,"totalJvmMemoryBytes":324534272,"freeJvmMemoryBytes":68595424,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":103760388,"jvmStartTime":1661681560910,"availableProcessors":6,"loadAverage":0.02,"totalInit":139788288,"totalUsed":474791104,"totalCommitted":557662208,"pid":1522332,"jvmName":"1522332@vm-1-cs-mgmt.aaaaaaaa.bbb
>>> > >
>>> >
>>> ","jvmVendor":"Ubuntu","jvmVersion":"11.0.16+8-post-Ubuntu-0ubuntu120.04","osDistribution":"Ubuntu
>>> > > > 20.04.5
>>> > > >
>>> > >
>>> >
>>> LTS","agentCount":5,"heapMemoryUsed":256775312,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":13,"threadsRunnableCount":15,"threadsTerminatedCount":0,"threadsTotalCount":420,"threadsWaitingCount":331,"systemMemoryTotal":8343236608,"systemMemoryFree":2623193088,"systemMemoryUsed":814808,"systemMemoryVirtualSize":7494184960,"logInfo":"/var/log/cloudstack/management/management-server.log
>>> > > > using: 74M\non disk /dev/vda2 mounted on / (14%
>>> > > >
>>> > >
>>> >
>>> full)","systemTotalCpuCycles":15870.180000000002,"systemLoadAverages":[0.02,0.03,0.0],"systemCyclesUsage":[4116569,2615101,2716460424],"dbLocal":false,"usageLocal":false,"systemBootTime":"Jul
>>> > > > 8, 2022, 2:35:55 AM","kernelVersion":"5.4.0-121-generic"}
>>> > > > 2022-08-29 17:02:01,723 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-4:ctx-7abb1b5d) (logid:2320950d) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:02:05,417 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (Vm-Operations-Cleanup-1:ctx-e741e208) (logid:3f5129e0) VM
>>> Operation
>>> > > Thread
>>> > > > Running
>>> > > > 2022-08-29 17:02:05,725 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-4:ctx-8e7f6766) (logid:710cb3b2) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:02:08,626 DEBUG [c.c.a.m.AgentManagerImpl]
>>> > > > (AgentManager-Handler-11:null) (logid:) Ping from Routing host
>>> > > > 6(srv-3-comp.aaaaaaaa.bbb)
>>> > > >
>>> > > > 2022-08-29 17:02:09,727 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-4:ctx-26a5ff07) (logid:09b3c89a) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:02:13,729 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-4:ctx-b4d4e374) (logid:9048c7a7) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:02:17,731 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-4:ctx-3db3d208) (logid:7bb73f7b) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:02:21,734 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-1:ctx-fca1d248) (logid:1ed5381a) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:02:25,737 DEBUG [o.a.c.h.HAManagerImpl]
>>> > > > (BackgroundTaskPollManager-1:ctx-984e2382) (logid:e6c3d724) HA
>>> health
>>> > > check
>>> > > > task is running...
>>> > > > 2022-08-29 17:02:25,759 DEBUG [c.c.s.StatsCollector]
>>> > > > (secstorage-1:ctx-64c610ed) (logid:6603cd38) Verifying image
>>> storage
>>> > [1].
>>> > > > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
>>> > > > 2022-08-29 17:02:25,763 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
>>> > > > (secstorage-1:ctx-64c610ed) (logid:6603cd38) Zone [1] is ready to
>>> > launch
>>> > > > secondary storage VM.
>>> > > > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Zone 1 is ready to
>>> > launch
>>> > > > console proxy
>>> > > > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Expand console proxy
>>> > > standby
>>> > > > capacity for zone xxxxxxxxxx
>>> > > > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Allocating console
>>> proxy
>>> > > > standby capacity for zone [1].
>>> > > > 2022-08-29 17:02:26,030 INFO [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Found a stopped
>>> console
>>> > > > proxy, starting it. Vm id : 85
>>> > > > 2022-08-29 17:02:26,031 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) start parameter
>>> value of
>>> > > > enterHardwareSetup == <very null> during processing of queued job
>>> > > > 2022-08-29 17:02:26,037 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Sync job-4289
>>> execution
>>> > on
>>> > > > object VmWorkJobQueue.85
>>> > > > 2022-08-29 17:02:26,167 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (AsyncJobMgr-Heartbeat-1:ctx-bba6b89d) (logid:f9aed0eb) Execute
>>> > > sync-queue
>>> > > > item: SyncQueueItemVO {id:2106, queueId: 2105, contentType:
>>> AsyncJob,
>>> > > > contentId: 4289, lastProcessMsid: 90520740759984,
>>> lastprocessNumber: 2,
>>> > > > lastProcessTime: Mon Aug 29 17:02:26 CEST 2022, created: Mon Aug 29
>>> > > > 17:02:26 CEST 2022}
>>> > > > 2022-08-29 17:02:26,167 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (AsyncJobMgr-Heartbeat-1:ctx-bba6b89d) (logid:f9aed0eb) Schedule
>>> queued
>>> > > > job-4289
>>> > > > 2022-08-29 17:02:26,172 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
>>> (logid:f919f9b3)
>>> > > Add
>>> > > > job-4289 into job monitoring
>>> > > > 2022-08-29 17:02:26,181 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
>>> (logid:ba10fec5)
>>> > > > Executing AsyncJobVO: {id:4289, userId: 1, accountId: 1,
>>> instanceType:
>>> > > > null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
>>> > > >
>>> > >
>>> >
>>> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAVXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
>>> > > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
>>> 0,
>>> > > > result: null, initMsid: 90520740759984, completeMsid: null,
>>> > lastUpdated:
>>> > > > null, lastPolled: null, created: Mon Aug 29 17:02:26 CEST 2022,
>>> > removed:
>>> > > > null}
>>> > > > 2022-08-29 17:02:26,181 DEBUG [c.c.v.VmWorkJobDispatcher]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
>>> (logid:ba10fec5)
>>> > > Run
>>> > > > VM work job: com.cloud.vm.VmWorkStart for VM 85, job origin: 3445
>>> > > > 2022-08-29 17:02:26,183 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Execute VM work job:
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
>>> > > > 2022-08-29 17:02:26,184 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) orchestrating VM start for 'v-85-VM'
>>> > > > com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
>>> > > > 2022-08-29 17:02:26,192 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
>>> > > > transited from [Stopped] to [Starting] with event [StartRequested].
>>> > VM's
>>> > > > original host: null, new host: null, host before state transition:
>>> null
>>> > > > 2022-08-29 17:02:26,192 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Successfully transitioned to start state for VM
>>> > instance
>>> > > > {id: "85", name: "v-85-VM", uuid:
>>> > "110d8620-c029-4ca9-9084-210099725a80",
>>> > > > type="ConsoleProxy"} reservation id =
>>> > > 24bf955f-96d8-4f4f-ae33-2cae2f3e0425
>>> > > > 2022-08-29 17:02:26,198 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts:
>>> null
>>> > > > 2022-08-29 17:02:26,200 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) VM start attempt #1
>>> > > > 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
>>> > > > 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Trying to allocate a host and storage pools from
>>> dc:1,
>>> > > > pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
>>> > > > 1073741824
>>> > > > 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
>>> > > > 2022-08-29 17:02:26,211 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
>>> > > > 2022-08-29 17:02:26,212 DEBUG [c.c.d.FirstFitPlanner]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Searching all possible resources under this Zone:
>>> 1
>>> > > > 2022-08-29 17:02:26,213 DEBUG [c.c.d.FirstFitPlanner]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Listing clusters in order of aggregate capacity,
>>> that
>>> > > have
>>> > > > (at least one host with) enough CPU and RAM capacity under this
>>> Zone: 1
>>> > > > 2022-08-29 17:02:26,216 DEBUG [c.c.d.FirstFitPlanner]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Removing from the clusterId list these clusters
>>> from
>>> > > avoid
>>> > > > set: []
>>> > > > 2022-08-29 17:02:26,221 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
>>> > > > 2022-08-29 17:02:26,222 INFO [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested
>>> with
>>> > > > Custom[UEFI] Boot Type false
>>> > > > 2022-08-29 17:02:26,223 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in
>>> dc: 1
>>> > > > pod:1 cluster:1
>>> > > > 2022-08-29 17:02:26,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4
>>> > hosts
>>> > > to
>>> > > > check for allocation: [Host {"id": "7", "name":
>>> > > "srv-5-comp.aaaaaaaa.bbb",
>>> > > > "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"},
>>> Host
>>> > > > {"id": "1", "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}, Host
>>> {"id":
>>> > > "6",
>>> > > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
>>> {"id":
>>> > > "5",
>>> > > > "name": "srv-4-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}]
>>> > > > 2022-08-29 17:02:26,231 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for
>>> allocation
>>> > > > after prioritization: [Host {"id": "7", "name":
>>> > > "srv-5-comp.aaaaaaaa.bbb",
>>> > > > "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"},
>>> Host
>>> > > > {"id": "1", "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}, Host
>>> {"id":
>>> > > "6",
>>> > > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
>>> {"id":
>>> > > "5",
>>> > > > "name": "srv-4-comp.aaaaaaaa.bbb", "uuid":
>>> > > > "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}]
>>> > > > 2022-08-29 17:02:26,231 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for
>>> speed=500Mhz,
>>> > > > Ram=1024 MB
>>> > > > 2022-08-29 17:02:26,231 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
>>> > > > srv-5-comp.aaaaaaaa.bbb, uuid:
>>> 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is
>>> > > KVM
>>> > > > hypervisor type, no max guest limit check needed
>>> > > > 2022-08-29 17:02:26,234 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu
>>> capability
>>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> speed:
>>> > 500
>>> > > > 2022-08-29 17:02:26,234 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has
>>> > enough
>>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > 1073741824 ,
>>> > > > cpuOverprovisioningFactor: 1.0
>>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> CPU:
>>> > > 127200
>>> > > > and CPU after applying overprovisioning: 127200
>>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 ,
>>> Requested
>>> > > CPU:
>>> > > > 500
>>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
>>> > > > 189373743104 , Requested RAM: (1.00 GB) 1073741824
>>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>>> RAM
>>> > > > available
>>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> from
>>> > > host:
>>> > > > 7, used: 90100, reserved: 0, actual total: 127200, total with
>>> > > > overprovisioning: 127200; requested
>>> cpu:500,alloc_from_last_host?:false
>>> > > > ,considerReservedCapacity?: true
>>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> from
>>> > > host:
>>> > > > 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total:
>>> (250.27
>>> > > GB)
>>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > alloc_from_last_host?:
>>> > > > false , considerReservedCapacity?: true
>>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>>> > adding
>>> > > to
>>> > > > list: 7
>>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
>>> > > > srv-2-comp.aaaaaaaa.bbb, uuid:
>>> 883f7f62-55f2-42f4-8242-16b6847c4204} is
>>> > > KVM
>>> > > > hypervisor type, no max guest limit check needed
>>> > > > 2022-08-29 17:02:26,239 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu
>>> capability
>>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> speed:
>>> > 500
>>> > > > 2022-08-29 17:02:26,239 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has
>>> > enough
>>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > 1073741824 ,
>>> > > > cpuOverprovisioningFactor: 1.0
>>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> CPU:
>>> > > 127200
>>> > > > and CPU after applying overprovisioning: 127200
>>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 ,
>>> Requested
>>> > > CPU:
>>> > > > 500
>>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
>>> > > > 229802639360 , Requested RAM: (1.00 GB) 1073741824
>>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>>> RAM
>>> > > > available
>>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> from
>>> > > host:
>>> > > > 1, used: 48200, reserved: 0, actual total: 127200, total with
>>> > > > overprovisioning: 127200; requested
>>> cpu:500,alloc_from_last_host?:false
>>> > > > ,considerReservedCapacity?: true
>>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> from
>>> > > host:
>>> > > > 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total:
>>> (250.27
>>> > > GB)
>>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > alloc_from_last_host?:
>>> > > > false , considerReservedCapacity?: true
>>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>>> > adding
>>> > > to
>>> > > > list: 1
>>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
>>> > > > srv-3-comp.aaaaaaaa.bbb, uuid:
>>> 6858713b-7e17-46d8-854d-24a13f9c7001} is
>>> > > KVM
>>> > > > hypervisor type, no max guest limit check needed
>>> > > > 2022-08-29 17:02:26,243 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu
>>> capability
>>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> speed:
>>> > 500
>>> > > > 2022-08-29 17:02:26,243 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has
>>> > enough
>>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > 1073741824 ,
>>> > > > cpuOverprovisioningFactor: 1.0
>>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> CPU:
>>> > > 127200
>>> > > > and CPU after applying overprovisioning: 127200
>>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 ,
>>> Requested
>>> > > CPU:
>>> > > > 500
>>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
>>> > > > 204032835584 , Requested RAM: (1.00 GB) 1073741824
>>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>>> RAM
>>> > > > available
>>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> from
>>> > > host:
>>> > > > 6, used: 42900, reserved: 0, actual total: 127200, total with
>>> > > > overprovisioning: 127200; requested
>>> cpu:500,alloc_from_last_host?:false
>>> > > > ,considerReservedCapacity?: true
>>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> from
>>> > > host:
>>> > > > 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total:
>>> (250.27
>>> > > GB)
>>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > alloc_from_last_host?:
>>> > > > false , considerReservedCapacity?: true
>>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>>> > adding
>>> > > to
>>> > > > list: 6
>>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
>>> > > > srv-4-comp.aaaaaaaa.bbb, uuid:
>>> 134c625d-15ab-461a-a38d-36fb0b979b5d} is
>>> > > KVM
>>> > > > hypervisor type, no max guest limit check needed
>>> > > > 2022-08-29 17:02:26,247 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu
>>> capability
>>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> speed:
>>> > 500
>>> > > > 2022-08-29 17:02:26,247 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has
>>> > enough
>>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > 1073741824 ,
>>> > > > cpuOverprovisioningFactor: 1.0
>>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> CPU:
>>> > > 127200
>>> > > > and CPU after applying overprovisioning: 127200
>>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 ,
>>> Requested
>>> > > > CPU: 500
>>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
>>> > > > 251009040384 , Requested RAM: (1.00 GB) 1073741824
>>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>>> RAM
>>> > > > available
>>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> from
>>> > > host:
>>> > > > 5, used: 11100, reserved: 0, actual total: 127200, total with
>>> > > > overprovisioning: 127200; requested
>>> cpu:500,alloc_from_last_host?:false
>>> > > > ,considerReservedCapacity?: true
>>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> from
>>> > > host:
>>> > > > 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total:
>>> (250.27
>>> > > GB)
>>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > alloc_from_last_host?:
>>> > > > false , considerReservedCapacity?: true
>>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>>> > adding
>>> > > to
>>> > > > list: 5
>>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator
>>> returning 4
>>> > > > suitable hosts
>>> > > > 2022-08-29 17:02:26,260 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
>>> > > > transited from [Starting] to [Stopped] with event
>>> [OperationFailed].
>>> > VM's
>>> > > > original host: null, new host: null, host before state transition:
>>> null
>>> > > > 2022-08-29 17:02:26,260 WARN [c.c.v.VirtualMachineManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Unable to orchestrate start VM instance {id: "85",
>>> > name:
>>> > > > "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
>>> > > > type="ConsoleProxy"} due to [Unable to create deployment, no usable
>>> > > volumes
>>> > > > found for the VM: 85].
>>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to create
>>> > > > deployment, no usable volumes found for the VM: 85
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
>>> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>>> > Source)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>>> > > > at
>>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> > > > at
>>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>>> > > > 2022-08-29 17:02:26,261 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Done executing VM work job:
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
>>> > > > 2022-08-29 17:02:26,262 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Complete async job-4289, jobStatus: FAILED,
>>> > resultCode:
>>> > > 0,
>>> > > > result:
>>> > > >
>>> > >
>>> >
>>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODUiLCBuYW1lOiAidi04NS1WTSIsIHV1aWQ6ICIxMTBkODYyMC1jMDI5LTRjYTktOTA4NC0yMTAwOTk3MjVhODAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODVdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
>>> > > > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Publish async job-4289 complete on message bus
>>> > > > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Wake up jobs related to job-4289
>>> > > > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Update db status for job-4289
>>> > > > 2022-08-29 17:02:26,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>>> > > > (logid:ba10fec5) Wake up jobs joined with job-4289 and disjoin all
>>> > > subjobs
>>> > > > created from job- 4289
>>> > > > 2022-08-29 17:02:26,269 DEBUG [c.c.v.VmWorkJobDispatcher]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
>>> (logid:ba10fec5)
>>> > > > Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 85,
>>> job
>>> > > > origin: 3445
>>> > > > 2022-08-29 17:02:26,270 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
>>> (logid:ba10fec5)
>>> > > > Done executing com.cloud.vm.VmWorkStart for job-4289
>>> > > > 2022-08-29 17:02:26,271 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
>>> (logid:ba10fec5)
>>> > > > Remove job-4289 from job monitoring
>>> > > > 2022-08-29 17:02:26,277 WARN [c.c.c.ConsoleProxyManagerImpl]
>>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Unable to allocate
>>> > console
>>> > > > proxy standby capacity for zone [1] due to [Unable to orchestrate
>>> start
>>> > > VM
>>> > > > instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>>> > > [Unable
>>> > > > to create deployment, no usable volumes found for the VM: 85].].
>>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to
>>> orchestrate
>>> > > > start VM instance {id: "85", name: "v-85-VM", uuid:
>>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>>> > > [Unable
>>> > > > to create deployment, no usable volumes found for the VM: 85].
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
>>> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>>> > Source)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>>> > > > at
>>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> > > > at
>>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> > > > at
>>> > > >
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>>> > > > 2022-08-29 17:02:26,277 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) received console
>>> proxy
>>> > > alert
>>> > > >
>>> > > >
>>> > > >
>>> > > > Am Mo., 29. Aug. 2022 um 16:30 Uhr schrieb Slavka Peleva <
>>> > > > slavkap@storpool.com>:
>>> > > >
>>> > > >> Thanks, Chris, for the information!
>>> > > >>
>>> > > >> Are you using custom offerings for the system VMs? Can you share
>>> more
>>> > > >> info if yes?
>>> > > >>
>>> > > >> About the log, it's not the complete log from where the deployment
>>> > > >> started. The deployment begins a bit earlier than what you
>>> shared. Can
>>> > > you
>>> > > >> search the management log for `Sync job-3539 execution on object
>>> > > >> VmWorkJobQueue` to get the `logid` for the volume creation
>>> (coloured
>>> > in
>>> > > >> blue)? And if possible, share the logs with the found log ID
>>> > > >>
>>> > > >> Here is an example from my logs:
>>> > > >>
>>> > > >> 2022-08-29 17:07:49,114 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > >> (consoleproxy-1:ctx-a952a3dc) (logid:89157bb9) Sync job-60484
>>> > execution
>>> > > >> on object VmWorkJobQueue.4904
>>> > > >>
>>> > > >> 2022-08-29 17:07:50,816 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>>> > > >> (Work-Job-Executor-5:ctx-56172581 job-60479/job-60484
>>> ctx-a900986d)
>>> > > >> (logid:95d750ae) Execute VM work job:
>>> > > >>
>>> > >
>>> >
>>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":4904,"handlerName":"VirtualMachineManagerImpl"}
>>> > > >> 2022-08-29 17:07:50,817 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > >> (Work-Job-Executor-5:ctx-56172581 job-60479/job-60484
>>> ctx-a900986d)
>>> > > >> (logid:95d750ae) orchestrating VM start for 'v-4904-VM'
>>> > > >> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
>>> > > >>
>>> > > >>
>>> > > >> On Mon, Aug 29, 2022 at 2:29 PM Vash_X@gmx.de <Va...@gmx.de>
>>> wrote:
>>> > > >>
>>> > > >>> Hi Slavka,
>>> > > >>>
>>> > > >>> thats what i did after your suggestion / informatin and where the
>>> > error
>>> > > >>> occured.
>>> > > >>> Deleted / Destroyed the consoleproxy and waited till recreation.
>>> > Which
>>> > > >>> didn't worked throwing the error i pasted above.
>>> > > >>> Sorry for beeing so unspecific.
>>> > > >>>
>>> > > >>> Regards,
>>> > > >>> Chris
>>> > > >>>
>>> > > >>> Am Mo., 29. Aug. 2022 um 12:46 Uhr schrieb Slavka Peleva <
>>> > > >>> slavkap@storpool.com>:
>>> > > >>>
>>> > > >>>> Hi Chris,
>>> > > >>>>
>>> > > >>>> I mean to recreate the system VMs by destroying them.
>>> > > >>>>
>>> > > >>>> On Mon, Aug 29, 2022 at 12:38 PM Vash_X@gmx.de <Va...@gmx.de>
>>> > wrote:
>>> > > >>>>
>>> > > >>>>> Hi Slavka,
>>> > > >>>>>
>>> > > >>>>> didn't tried to restart the SystemVMs. Reading through the
>>> docs i
>>> > had
>>> > > >>>>> the impression that "live-patching" would be enough for
>>> this....
>>> > > >>>>> Anyway - I tried my luck with the console proxy.
>>> > > >>>>>
>>> > > >>>>> Which failed utterly. Not starting as it isn't finding any
>>> useable
>>> > > >>>>> storage pool. - logs at the end of the mail.
>>> > > >>>>>
>>> > > >>>>> I tried deploying a VM using the same storage pool which worked
>>> > > >>>>> wonderfully.
>>> > > >>>>> There i noticed that all my custome storage-offerings are
>>> unuseable
>>> > > at
>>> > > >>>>> the moment. They are not accepting any custome storage sizes
>>> > > anymore...
>>> > > >>>>> Throwing failures "VM Creation failed. Volume size: -1GB is
>>> out of
>>> > > allowed
>>> > > >>>>> tange. Max:3072 Min:1"...
>>> > > >>>>> So this is also currently broken....
>>> > > >>>>>
>>> > > >>>>> 2022-08-29 10:51:56,189 DEBUG [c.c.v.VmWorkJobDispatcher]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
>>> > > (logid:ba10fec5) Run
>>> > > >>>>> VM work job: com.cloud.vm.VmWorkStart for VM 83, job origin:
>>> 3445
>>> > > >>>>> 2022-08-29 10:51:56,191 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Execute VM work job:
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
>>> > > >>>>> 2022-08-29 10:51:56,192 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) orchestrating VM start for 'v-83-VM'
>>> > > >>>>> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
>>> > > >>>>> 2022-08-29 10:51:56,198 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
>>> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
>>> state
>>> > > >>>>> transited from [Stopped] to [Starting] with event
>>> [StartRequested].
>>> > > VM's
>>> > > >>>>> original host: null, new host: null, host before state
>>> transition:
>>> > > null
>>> > > >>>>> 2022-08-29 10:51:56,198 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Successfully transitioned to start state for
>>> VM
>>> > > instance
>>> > > >>>>> {id: "83", name: "v-83-VM", uuid:
>>> > > "b1d1f9c6-eafe-4527-9335-e649c646aab0",
>>> > > >>>>> type="ConsoleProxy"} reservation id =
>>> > > 6641ccc0-bcef-4b1a-a1f7-8dd8000626ad
>>> > > >>>>> 2022-08-29 10:51:56,203 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Deploy avoids pods: null, clusters: null,
>>> hosts:
>>> > > null
>>> > > >>>>> 2022-08-29 10:51:56,206 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) VM start attempt #1
>>> > > >>>>> 2022-08-29 10:51:56,207 DEBUG
>>> [c.c.d.DeploymentPlanningManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
>>> > > >>>>> 2022-08-29 10:51:56,207 DEBUG
>>> [c.c.d.DeploymentPlanningManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Trying to allocate a host and storage pools
>>> from
>>> > > dc:1,
>>> > > >>>>> pod:null,cluster:null, requested cpu: 500, requested ram:
>>> (1.00 GB)
>>> > > >>>>> 1073741824
>>> > > >>>>> 2022-08-29 10:51:56,207 DEBUG
>>> [c.c.d.DeploymentPlanningManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Is ROOT volume READY (pool already
>>> allocated)?: No
>>> > > >>>>> 2022-08-29 10:51:56,214 DEBUG
>>> [c.c.d.DeploymentPlanningManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts:
>>> null
>>> > > >>>>> 2022-08-29 10:51:56,215 DEBUG [c.c.d.FirstFitPlanner]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Searching all possible resources under this
>>> Zone:
>>> > 1
>>> > > >>>>> 2022-08-29 10:51:56,216 DEBUG [c.c.d.FirstFitPlanner]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Listing clusters in order of aggregate
>>> capacity,
>>> > > that have
>>> > > >>>>> (at least one host with) enough CPU and RAM capacity under this
>>> > > Zone: 1
>>> > > >>>>> 2022-08-29 10:51:56,219 DEBUG [c.c.d.FirstFitPlanner]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Removing from the clusterId list these
>>> clusters
>>> > > from avoid
>>> > > >>>>> set: []
>>> > > >>>>> 2022-08-29 10:51:56,224 DEBUG
>>> [c.c.d.DeploymentPlanningManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
>>> > > >>>>> 2022-08-29 10:51:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is
>>> requested
>>> > > with
>>> > > >>>>> Custom[UEFI] Boot Type false
>>> > > >>>>> 2022-08-29 10:51:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in
>>> > dc: 1
>>> > > >>>>> pod:1 cluster:1
>>> > > >>>>> 2022-08-29 10:51:56,227 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator
>>> has 4
>>> > > hosts to
>>> > > >>>>> check for allocation: [Host {"id": "7", "name":
>>> > > "srv-5-comp.blabla.blub",
>>> > > >>>>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6",
>>> "type"="Routing"},
>>> > > Host
>>> > > >>>>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
>>> > > >>>>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
>>> > > {"id": "6",
>>> > > >>>>> "name": "srv-3-comp.blabla.blub", "uuid":
>>> > > >>>>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
>>> > > {"id": "1",
>>> > > >>>>> "name": "srv-2-comp.blabla.blub", "uuid":
>>> > > >>>>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
>>> > > >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for
>>> > > allocation
>>> > > >>>>> after prioritization: [Host {"id": "7", "name":
>>> > > "srv-5-comp.blabla.blub",
>>> > > >>>>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6",
>>> "type"="Routing"},
>>> > > Host
>>> > > >>>>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
>>> > > >>>>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
>>> > > {"id": "6",
>>> > > >>>>> "name": "srv-3-comp.blabla.blub", "uuid":
>>> > > >>>>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
>>> > > {"id": "1",
>>> > > >>>>> "name": "srv-2-comp.blabla.blub", "uuid":
>>> > > >>>>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
>>> > > >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for
>>> > speed=500Mhz,
>>> > > >>>>> Ram=1024 MB
>>> > > >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
>>> > > >>>>> srv-5-comp.blabla.blub, uuid:
>>> 8c753ce1-0253-49c9-9c7f-1f971a719ce6}
>>> > > is KVM
>>> > > >>>>> hypervisor type, no max guest limit check needed
>>> > > >>>>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu
>>> > capability
>>> > > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> > > speed: 500
>>> > > >>>>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7
>>> has
>>> > > enough
>>> > > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > > 1073741824 ,
>>> > > >>>>> cpuOverprovisioningFactor: 1.0
>>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> > CPU:
>>> > > 127200
>>> > > >>>>> and CPU after applying overprovisioning: 127200
>>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 ,
>>> > > Requested CPU:
>>> > > >>>>> 500
>>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37
>>> GB)
>>> > > >>>>> 189373743104 , Requested RAM: (1.00 GB) 1073741824
>>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
>>> and
>>> > > RAM
>>> > > >>>>> available
>>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> > from
>>> > > host:
>>> > > >>>>> 7, used: 90100, reserved: 0, actual total: 127200, total with
>>> > > >>>>> overprovisioning: 127200; requested
>>> > > cpu:500,alloc_from_last_host?:false
>>> > > >>>>> ,considerReservedCapacity?: true
>>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> > from
>>> > > host:
>>> > > >>>>> 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total:
>>> > > (250.27 GB)
>>> > > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > > alloc_from_last_host?:
>>> > > >>>>> false , considerReservedCapacity?: true
>>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable
>>> host,
>>> > > adding to
>>> > > >>>>> list: 7
>>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
>>> > > >>>>> srv-4-comp.blabla.blub, uuid:
>>> 134c625d-15ab-461a-a38d-36fb0b979b5d}
>>> > > is KVM
>>> > > >>>>> hypervisor type, no max guest limit check needed
>>> > > >>>>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu
>>> > capability
>>> > > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> > > speed: 500
>>> > > >>>>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5
>>> has
>>> > > enough
>>> > > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > > 1073741824 ,
>>> > > >>>>> cpuOverprovisioningFactor: 1.0
>>> > > >>>>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> > CPU:
>>> > > 127200
>>> > > >>>>> and CPU after applying overprovisioning: 127200
>>> > > >>>>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 ,
>>> > > Requested
>>> > > >>>>> CPU: 500
>>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77
>>> GB)
>>> > > >>>>> 251009040384 , Requested RAM: (1.00 GB) 1073741824
>>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
>>> and
>>> > > RAM
>>> > > >>>>> available
>>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> > from
>>> > > host:
>>> > > >>>>> 5, used: 11100, reserved: 0, actual total: 127200, total with
>>> > > >>>>> overprovisioning: 127200; requested
>>> > > cpu:500,alloc_from_last_host?:false
>>> > > >>>>> ,considerReservedCapacity?: true
>>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> > from
>>> > > host:
>>> > > >>>>> 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total:
>>> > > (250.27 GB)
>>> > > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > > alloc_from_last_host?:
>>> > > >>>>> false , considerReservedCapacity?: true
>>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable
>>> host,
>>> > > adding to
>>> > > >>>>> list: 5
>>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
>>> > > >>>>> srv-3-comp.blabla.blub, uuid:
>>> 6858713b-7e17-46d8-854d-24a13f9c7001}
>>> > > is KVM
>>> > > >>>>> hypervisor type, no max guest limit check needed
>>> > > >>>>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu
>>> > capability
>>> > > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> > > speed: 500
>>> > > >>>>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6
>>> has
>>> > > enough
>>> > > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > > 1073741824 ,
>>> > > >>>>> cpuOverprovisioningFactor: 1.0
>>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> > CPU:
>>> > > 127200
>>> > > >>>>> and CPU after applying overprovisioning: 127200
>>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 ,
>>> > > Requested CPU:
>>> > > >>>>> 500
>>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02
>>> GB)
>>> > > >>>>> 204032835584 , Requested RAM: (1.00 GB) 1073741824
>>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
>>> and
>>> > > RAM
>>> > > >>>>> available
>>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> > from
>>> > > host:
>>> > > >>>>> 6, used: 42900, reserved: 0, actual total: 127200, total with
>>> > > >>>>> overprovisioning: 127200; requested
>>> > > cpu:500,alloc_from_last_host?:false
>>> > > >>>>> ,considerReservedCapacity?: true
>>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> > from
>>> > > host:
>>> > > >>>>> 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total:
>>> > > (250.27 GB)
>>> > > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > > alloc_from_last_host?:
>>> > > >>>>> false , considerReservedCapacity?: true
>>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable
>>> host,
>>> > > adding to
>>> > > >>>>> list: 6
>>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
>>> > > >>>>> srv-2-comp.blabla.blub, uuid:
>>> 883f7f62-55f2-42f4-8242-16b6847c4204}
>>> > > is KVM
>>> > > >>>>> hypervisor type, no max guest limit check needed
>>> > > >>>>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu
>>> > capability
>>> > > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
>>> > > speed: 500
>>> > > >>>>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1
>>> has
>>> > > enough
>>> > > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>>> > > 1073741824 ,
>>> > > >>>>> cpuOverprovisioningFactor: 1.0
>>> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>>> > CPU:
>>> > > 127200
>>> > > >>>>> and CPU after applying overprovisioning: 127200
>>> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 ,
>>> > > Requested CPU:
>>> > > >>>>> 500
>>> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02
>>> GB)
>>> > > >>>>> 229802639360 , Requested RAM: (1.00 GB) 1073741824
>>> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
>>> and
>>> > > RAM
>>> > > >>>>> available
>>> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>>> > from
>>> > > host:
>>> > > >>>>> 1, used: 48200, reserved: 0, actual total: 127200, total with
>>> > > >>>>> overprovisioning: 127200; requested
>>> > > cpu:500,alloc_from_last_host?:false
>>> > > >>>>> ,considerReservedCapacity?: true
>>> > > >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>>> > from
>>> > > host:
>>> > > >>>>> 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total:
>>> > > (250.27 GB)
>>> > > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
>>> > > alloc_from_last_host?:
>>> > > >>>>> false , considerReservedCapacity?: true
>>> > > >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable
>>> host,
>>> > > adding to
>>> > > >>>>> list: 1
>>> > > >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751
>>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator
>>> > returning 4
>>> > > >>>>> suitable hosts
>>> > > >>>>> 2022-08-29 10:51:56,260 DEBUG [c.c.c.CapacityManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
>>> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
>>> state
>>> > > >>>>> transited from [Starting] to [Stopped] with event
>>> > [OperationFailed].
>>> > > VM's
>>> > > >>>>> original host: null, new host: null, host before state
>>> transition:
>>> > > null
>>> > > >>>>> 2022-08-29 10:51:56,260 WARN [c.c.v.VirtualMachineManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Unable to orchestrate start VM instance {id:
>>> "83",
>>> > > name:
>>> > > >>>>> "v-83-VM", uuid: "b1d1f9c6-eafe-4527-9335-e649c646aab0",
>>> > > >>>>> type="ConsoleProxy"} due to [Unable to create deployment, no
>>> usable
>>> > > volumes
>>> > > >>>>> found for the VM: 83].
>>> > > >>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to
>>> create
>>> > > >>>>> deployment, no usable volumes found for the VM: 83
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
>>> > > >>>>> at
>>> jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>>> > > >>>>> Source)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > > >>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>>> > > >>>>> at
>>> > > >>>>>
>>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> > > >>>>> at
>>> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> > > >>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>> > > >>>>> 2022-08-29 10:51:56,262 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Done executing VM work job:
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
>>> > > >>>>> 2022-08-29 10:51:56,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Complete async job-3539, jobStatus: FAILED,
>>> > > resultCode: 0,
>>> > > >>>>> result:
>>> > > >>>>>
>>> > >
>>> >
>>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODMiLCBuYW1lOiAidi04My1WTSIsIHV1aWQ6ICJiMWQxZjljNi1lYWZlLTQ1MjctOTMzNS1lNjQ5YzY0NmFhYjAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODNdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
>>> > > >>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Publish async job-3539 complete on message bus
>>> > > >>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Wake up jobs related to job-3539
>>> > > >>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Update db status for job-3539
>>> > > >>>>> 2022-08-29 10:51:56,266 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>>> ctx-073f2751)
>>> > > >>>>> (logid:ba10fec5) Wake up jobs joined with job-3539 and disjoin
>>> all
>>> > > subjobs
>>> > > >>>>> created from job- 3539
>>> > > >>>>> 2022-08-29 10:51:56,276 DEBUG [c.c.v.VmWorkJobDispatcher]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
>>> > > (logid:ba10fec5) Done
>>> > > >>>>> with run of VM work job: com.cloud.vm.VmWorkStart for VM 83,
>>> job
>>> > > origin:
>>> > > >>>>> 3445
>>> > > >>>>> 2022-08-29 10:51:56,276 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
>>> > > (logid:ba10fec5) Done
>>> > > >>>>> executing com.cloud.vm.VmWorkStart for job-3539
>>> > > >>>>> 2022-08-29 10:51:56,278 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
>>> > > (logid:ba10fec5)
>>> > > >>>>> Remove job-3539 from job monitoring
>>> > > >>>>> 2022-08-29 10:51:56,285 WARN [c.c.c.ConsoleProxyManagerImpl]
>>> > > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Unable to
>>> allocate
>>> > > console
>>> > > >>>>> proxy standby capacity for zone [1] due to [Unable to
>>> orchestrate
>>> > > start VM
>>> > > >>>>> instance {id: "83", name: "v-83-VM", uuid:
>>> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
>>> due to
>>> > > [Unable
>>> > > >>>>> to create deployment, no usable volumes found for the VM:
>>> 83].].
>>> > > >>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to
>>> > > orchestrate
>>> > > >>>>> start VM instance {id: "83", name: "v-83-VM", uuid:
>>> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
>>> due to
>>> > > [Unable
>>> > > >>>>> to create deployment, no usable volumes found for the VM: 83].
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
>>> > > >>>>> at
>>> jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>>> > > >>>>> Source)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> > > >>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>>> > > >>>>> at
>>> > > >>>>>
>>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> > > >>>>> at
>>> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> > > >>>>> at
>>> > > >>>>>
>>> > >
>>> >
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> > > >>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>> > > >>>>> 2022-08-29 10:51:56,287 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>>> > > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) received console
>>> > > proxy alert
>>> > > >>>>> 2022-08-29 10:51:56,288 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>>> > > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Console proxy
>>> > creation
>>> > > >>>>> failure, zone: blabla
>>> > > >>>>> 2022-08-29 10:51:56,289 WARN [c.c.a.AlertManagerImpl]
>>> > > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) alertType=[10]
>>> > > >>>>> dataCenterId=[1] podId=[null] clusterId=[null] message=[Console
>>> > proxy
>>> > > >>>>> creation failure. zone: blabla, error details: Unable to
>>> > orchestrate
>>> > > start
>>> > > >>>>> VM instance {id: "83", name: "v-83-VM", uuid:
>>> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
>>> due to
>>> > > [Unable
>>> > > >>>>> to create deployment, no usable volumes found for the VM:
>>> 83].].
>>> > > >>>>>
>>> > > >>>>>
>>> > > >>>>>
>>> > > >>>>>
>>> > > >>>>> Am Mo., 29. Aug. 2022 um 09:40 Uhr schrieb Slavka Peleva
>>> > > >>>>> <sl...@storpool.com.invalid>:
>>> > > >>>>>
>>> > > >>>>>> Hi Chris,
>>> > > >>>>>>
>>> > > >>>>>> Did you recreate the system VMs? In the 4.17 version, the
>>> > > >>>>>> `systemvm.iso` is
>>> > > >>>>>> deprecated. In its place, you should have the file on agents
>>> > > >>>>>> `/usr/share/cloudstack-common/vms/agent.zip`.
>>> > > >>>>>> Can you share the complete log if the system VMs are newly
>>> created
>>> > > >>>>>> after
>>> > > >>>>>> the upgrade?
>>> > > >>>>>>
>>> > > >>>>>> Best regards,
>>> > > >>>>>> Slavka
>>> > > >>>>>>
>>> > > >>>>>> On Sun, Aug 28, 2022 at 2:16 PM Vash_X@gmx.de <Va...@gmx.de>
>>> > > wrote:
>>> > > >>>>>>
>>> > > >>>>>> > Hi everyone,
>>> > > >>>>>> >
>>> > > >>>>>> > faceing some challanges again after upgradeing to 4.17.
>>> > > >>>>>> > Did as explained in the docs.
>>> > > >>>>>> >
>>> > > >>>>>> > Afterwards I am now not able to migrate system VMs to
>>> different
>>> > > >>>>>> hosts.
>>> > > >>>>>> > Getting the error
>>> > > >>>>>> > "Exception during migrate: org.libvirt.LibvirtException:
>>> Cannot
>>> > > >>>>>> access
>>> > > >>>>>> > storage file
>>> '/usr/share/cloudstack-common/vms/systemvm.iso': No
>>> > > >>>>>> such file
>>> > > >>>>>> > or directory"
>>> > > >>>>>> >
>>> > > >>>>>> > Checked the exitstence of the file on the servers:
>>> > > >>>>>> > Mangement-Server has a "systemvm.iso.bak"
>>> > > >>>>>> > Hosts - no "systemvm.iso" at all present.
>>> > > >>>>>> >
>>> > > >>>>>> > Maybe someone has an idea on how to move on with this?
>>> > > >>>>>> >
>>> > > >>>>>> > Regards,
>>> > > >>>>>> > Chris
>>> > > >>>>>> >
>>> > > >>>>>>
>>> > > >>>>>
>>> > >
>>> >
>>>
>>
Re: Error migrate System VMs after upgrageing to 4.17
Posted by "Vash_X@gmx.de" <Va...@gmx.de>.
Hi Slavka,
be prepared vor a longer answer as this seems to be a more specific problem
- but first let me thank you for your guidance and support!
> The discussion started about a problem migrating the system VMs, and
probably many other issues will appear.
Don't give up :-D I might see some light ahead....
Bevore getting into the logs and my findings let me answer you your
questions briefly.
> Can you tell from which version you started the upgrade?
I started from 4.16.1; the installation was builded with 4.15.1 and then
updated to 4.16 / 4.16.1
> Did the upgrade was successful for the management/agents?
The upgrade was successfull - even if we had our issues which are currently
in a ticket at github https://github.com/apache/cloudstack/issues/6680
The upgrade of the management server was successfully performed too.
> Did you restart the agents on all machines?
We did as we had trouble with the changed detection of CPU speeds on the
host.
The whole "problem" of the SystemVM.iso is at least from my point of view
no problem of the agent-update.
a) we had the expection from reading the upgradeguide, that a "simple"
patching of the systemVMs would be enough. That at least was our
understanding of the "matrix" showing the patching capabilities of the
different versions / "upgrade paths".
b) currently we assume, that the main-reason for the failing migration is,
that the SSVM / CPVM are still running without a clear "restart". Which we
can't test as we still have to solve the problem of the public IP
allocation.
> Can you share the content in the directory
`/usr/share/cloudstack-agent/lib/` from the hypervisor where the migration
failed?
activation-1.1.1.jar
cloud-framework-events-4.17.0.1.jar
error_prone_annotations-2.3.4.jar jaxb-impl-2.3.0.jar
okio-2.8.0.jar
annotations-13.0.jar
cloud-framework-ipc-4.17.0.1.jar esapi-2.1.0.1.jar
jbcrypt-1.0.0.jar
opensaml-2.6.6.jar
antisamy-1.5.3.jar
cloud-framework-jobs-4.17.0.1.jar failureaccess-1.0.1.jar
jcl-over-slf4j-1.7.5.jar
openws-1.5.6.jar
aopalliance-repackaged-2.5.0-b42.jar
cloud-framework-managed-context-4.17.0.1.jar groovy-all-2.4.17.jar
jersey-client-2.26.jar
osgi-resource-locator-1.0.1.jar
apache-log4j-extras-1.2.17.jar
cloud-framework-rest-4.17.0.1.jar gson-1.7.2.jar
jersey-common-2.26.jar
protobuf-java-3.11.1.jar
aspectjweaver-1.8.13.jar
cloud-framework-security-4.17.0.1.jar guava-30.1-jre.jar
jersey-entity-filtering-2.26.jar
rados-0.6.0.jar
aws-java-sdk-core-1.11.940.jar
cloud-plugin-hypervisor-kvm-4.17.0.1.jar guava-testlib-18.0.jar
jersey-hk2-2.26.jar
reflections-0.9.12.jar
aws-java-sdk-kms-1.11.940.jar
cloud-plugin-network-ovs-4.17.0.1.jar hk2-api-2.5.0-b42.jar
jersey-media-json-jackson-2.26.jar
reload4j-1.2.20.jar
aws-java-sdk-s3-1.11.940.jar
cloud-plugin-storage-volume-scaleio-4.17.0.1.jar hk2-locator-2.5.0-b42.jar
jersey-media-multipart-2.26.jar
retrofit-2.9.0.jar
batik-constants-1.14.jar
cloud-secondary-storage-4.17.0.1.jar hk2-utils-2.5.0-b42.jar
jmespath-java-1.11.940.jar
rolling-maintenance
batik-css-1.14.jar cloud-server-4.17.0.1.jar
httpclient-4.5.13.jar
jna-5.5.0.jar
slf4j-api-1.7.5.jar
batik-i18n-1.14.jar cloud-utils-4.17.0.1.jar
httpcore-4.4.14.jar
jna-platform-5.5.0.jar
spring-aop-5.3.18.jar
batik-shared-resources-1.14.jar commons-beanutils-1.9.2.jar
influxdb-java-2.21.jar
joda-time-2.10.9.jar
spring-beans-5.3.18.jar
batik-util-1.14.jar commons-codec-1.15.jar
ini4j-0.5.4.jar
jsch-0.1.55.jar
spring-context-5.3.18.jar
bcpkix-jdk15on-1.70.jar
commons-collections-3.2.1.jar ion-java-1.0.2.jar
json-20090211.jar
spring-core-5.3.18.jar
bcprov-jdk15on-1.70.jar commons-compress-1.21.jar
j2objc-annotations-1.3.jar
jsr305-1.3.9.jar
spring-expression-5.3.18.jar
bctls-jdk15on-1.70.jar
commons-configuration-1.10.jar
jackson-annotations-2.12.5.jar jul-to-slf4j-1.7.5.jar
spring-jcl-5.3.18.jar
bcutil-jdk15on-1.70.jar commons-daemon-1.2.3.jar
jackson-core-2.12.5.jar
kotlin-stdlib-1.4.10.jar
spring-web-5.3.18.jar
bsh-core-2.0b4.jar commons-dbcp2-2.8.0.jar
jackson-databind-2.12.5.jar
kotlin-stdlib-common-1.4.0.jar
stax2-api-3.1.4.jar
cglib-nodep-3.3.0.jar commons-digester-1.8.1.jar
jackson-dataformat-cbor-2.6.7.jar
kotlin-stdlib-jdk7-1.4.10.jar
swagger-annotations-2.0.0.jar
checker-qual-3.5.0.jar commons-email-1.5.jar
jackson-datatype-jsr310-2.12.5.jar
kotlin-stdlib-jdk8-1.4.10.jar
tink-1.5.0.jar
cloud-agent-4.17.0.1.jar commons-fileupload-1.4.jar
jackson-jaxrs-base-2.12.1.jar
libvirt-0.5.2.jar
trilead-ssh2-build-217-jenkins-27.jar
cloud-api-4.17.0.1.jar commons-httpclient-3.1.jar
jackson-jaxrs-json-provider-2.12.1.jar
libvirtqemuhook
truth-0.23.jar
cloud-core-4.17.0.1.jar commons-io-2.8.0.jar
jackson-module-jaxb-annotations-2.8.4.jar
listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
velocity-1.7.jar
cloud-engine-api-4.17.0.1.jar commons-lang-2.6.jar
jasypt-1.9.3.jar
log4j-over-slf4j-1.7.5.jar
woodstox-core-5.0.3.jar
cloud-engine-components-api-4.17.0.1.jar commons-lang3-3.11.jar
java-ipv6-0.17.jar
logging-interceptor-4.9.0.jar
xalan-2.7.0.jar
cloud-engine-schema-4.17.0.1.jar commons-logging-1.2.jar
java-linstor-0.3.0.jar
mail-1.5.0-b01.jar
xercesImpl-2.8.0.jar
cloud-engine-storage-4.17.0.1.jar commons-math3-3.6.1.jar
javassist-3.27.0-GA.jar
metrics-core-3.0.2.jar
xmlgraphics-commons-2.6.jar
cloud-engine-storage-configdrive-4.17.0.1.jar commons-net-3.7.2.jar
javax.annotation-api-1.3.2.jar
metrics-jvm-3.0.2.jar
xmlpull-1.1.3.1.jar
cloud-engine-storage-image-4.17.0.1.jar commons-pool2-2.9.0.jar
javax.inject-1.jar
mimepull-1.9.6.jar
xmlschema-core-2.2.5.jar
cloud-engine-storage-volume-4.17.0.1.jar commons-validator-1.6.jar
javax.inject-2.5.0-b42.jar
moshi-1.8.0.jar
xmlsec-1.5.7.jar
cloud-framework-agent-lb-4.17.0.1.jar converter-moshi-2.9.0.jar
javax.mail-1.5.6.jar
msgpack-core-0.8.21.jar
xmltooling-1.4.6.jar
cloud-framework-ca-4.17.0.1.jar cxf-core-3.2.14.jar
javax.persistence-2.2.1.jar
mxparser-1.2.2.jar
xom-1.2.5.jar
cloud-framework-cluster-4.17.0.1.jar
cxf-rt-frontend-jaxrs-3.2.14.jar
javax.servlet-api-4.0.1.jar nekohtml-1.9.16.jar
xstream-1.4.19.jar
cloud-framework-config-4.17.0.1.jar
cxf-rt-transports-http-3.2.14.jar javax.ws.rs-api-2.1.jar
netty-all-4.1.42.Final.jar
cloud-framework-db-4.17.0.1.jar eddsa-0.3.0.jar
jaxb-api-2.3.0.jar
not-yet-commons-ssl-0.3.9.jar
cloud-framework-direct-download-4.17.0.1.jar ehcache-core-2.6.11.jar
jaxb-core-2.3.0.jar
And now to the problem of the ip's....
Thanks for your guidance, i guess i can give some more information. For the
findings to understand i guess i need to elaborate a bit on the current
configuration.
Currently we are having only one Zone up and running (ID=1).
In this Zone we have 3 different domains which are "Root","A","B".
Each domain has a account linked: Root = Admin / A = AccA / B = AccB.
The public network of the zone has currently 3 ip ranges which are: [Root]
system (System VMs = true) / [Root] / [B]AccB (written like in the WebGUI
which is i guess going to be important for understanding).
This kind of configuration has been in place since the initial setup and
has been working without problems.
Afterwards you will find the logs with the database querys - i started with
a deleting the stopped consoleproxy. i will only share the part when the
process of configuring the NIC / IPs for the console proxy will take place.
I put some statements into it - for me to get a better understanding of the
process - and i hope that you will get through this a bit better. The
comments are referenceing on the following "closing" statements. How ever i
didn't deleted anything.
###################################################################### Nic
allocation
2022-08-31 18:44:21,203 DEBUG [c.c.v.VirtualMachineManagerImpl]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Allocating nics for VM
instance {id: "86", name: "v-86-VM", uuid:
"6f77acb4-c65e-49e8-ae03-1175b3974412", type="ConsoleProxy"}
2022-08-31 18:44:21,203 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
consoleproxy-1 : tx-79, consoleproxy-1,
2022-08-31 18:44:21,204 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: start requested by:
-Transaction.execute:39-NetworkOrchestrator.allocate:778-VirtualMachineManagerImpl$1.doInTransactionWithoutResult:478-TransactionCallbackWithExceptionNoReturn.doInTransaction:25-TransactionCallbackWithExceptionNoReturn.doInTransaction:21-Transaction.execute:40-VirtualMachineManagerImpl.allocate:469-VirtualMachineManagerImpl.allocate:528-ConsoleProxyManagerImpl.createProxyInstance:700-ConsoleProxyManagerImpl.startNew:572-ConsoleProxyManagerImpl.allocCapacity:809-ConsoleProxyManagerImpl.expandPool:1537
2022-08-31 18:44:21,204 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: has already been
started.
2022-08-31 18:44:21,204 TRACE [o.a.c.e.o.NetworkOrchestrator]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) allocating networks for
v-86-VM(template c20799e4-a990-449a-aae6-4b67374c7aff); 3 networks
2022-08-31 18:44:21,204 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,206 TRACE [c.c.u.d.T.Connection]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Creating a DB connection
with txn: for 0: dbconn623913035. Stack:
-TransactionLegacy.prepareStatement:466-TransactionLegacy.prepareAutoCloseStatement:459-GenericDaoBase.searchIncludingRemoved:396-GenericDaoBase.searchIncludingRemoved:359-GenericDaoBase.search:348-GenericDaoBase.search:1319-TemplateDeployAsIsDetailsDaoImpl.listDetailsByTemplateIdMatchingPrefix:64-TemplateDeployAsIsDetailsDaoImpl.listNetworkRequirementsByTemplateId:69-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62-DelegatingMethodAccessorImpl.invoke:43-Method.invoke:566
2022-08-31 18:44:21,206 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
template_deploy_as_is_details.id,
template_deploy_as_is_details.template_id,
template_deploy_as_is_details.name, template_deploy_as_is_details.value
FROM template_deploy_as_is_details WHERE
template_deploy_as_is_details.template_id = ? AND
template_deploy_as_is_details.name LIKE ?
#### template_deploy_as_is_details = empty --> no results returned
2022-08-31 18:44:21,206 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
com.mysql.cj.jdbc.ClientPreparedStatement: SELECT
template_deploy_as_is_details.id,
template_deploy_as_is_details.template_id,
template_deploy_as_is_details.name, template_deploy_as_is_details.value
FROM template_deploy_as_is_details WHERE
template_deploy_as_is_details.template_id = 213 AND
template_deploy_as_is_details.name LIKE x'6E6574776F726B2D25'
2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Connection]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: Not closing DB
connection because we're still in a transaction.
2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
connection
2022-08-31 18:44:21,207 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT networks.id,
networks.mode, networks.broadcast_domain_type, networks.traffic_type,
networks.name, networks.display_text, networks.broadcast_uri,
networks.gateway, networks.cidr, networks.network_cidr,
networks.network_offering_id, networks.vpc_id,
networks.physical_network_id, networks.data_center_id, networks.related,
networks.guru_name, networks.state, networks.redundant, networks.dns1,
networks.domain_id, networks.account_id, networks.set_fields,
networks.guru_data, networks.dns2, networks.network_domain,
networks.removed, networks.created, networks.reservation_id, networks.uuid,
networks.guest_type, networks.acl_type, networks.restart_required,
networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr,
networks.display_network, networks.network_acl_id, networks.streched_l2,
networks.external_id FROM networks WHERE networks.id = ? AND
networks.removed IS NULL
2022-08-31 18:44:21,209 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Allocating nic for vm VM
instance {id: "86", name: "v-86-VM", uuid:
"6f77acb4-c65e-49e8-ae03-1175b3974412", type="ConsoleProxy"} in network
Network {"id": 200, "name": "null", "uuid":
"1b0bf965-a391-478d-833d-3e37e4a15eb8", "networkofferingid": 1} with
requested profile NicProfile
{"broadcastUri":null,"iPv4Address":null,"id":0,"reservationId":null,"vmId":0}
2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
connection
2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
data_center.id, data_center.name, data_center.description,
data_center.dns1, data_center.dns2, data_center.ip6_dns1,
data_center.ip6_dns2, data_center.internal_dns1, data_center.internal_dns2,
data_center.router_mac_address, data_center.guest_network_cidr,
data_center.domain_id, data_center.domain, data_center.networktype,
data_center.dns_provider, data_center.dhcp_provider,
data_center.gateway_provider, data_center.vpn_provider,
data_center.userdata_provider, data_center.lb_provider,
data_center.firewall_provider, data_center.mac_address,
data_center.zone_token, data_center.removed, data_center.allocation_state,
data_center.uuid, data_center.is_security_group_enabled,
data_center.is_local_storage_enabled, data_center.sort_key FROM data_center
WHERE data_center.id = ? AND data_center.removed IS NULL
### Allocating public nic
2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
com.mysql.cj.jdbc.ClientPreparedStatement: SELECT networks.id,
networks.mode, networks.broadcast_domain_type, networks.traffic_type,
networks.name, networks.display_text, networks.broadcast_uri,
networks.gateway, networks.cidr, networks.network_cidr,
networks.network_offering_id, networks.vpc_id,
networks.physical_network_id, networks.data_center_id, networks.related,
networks.guru_name, networks.state, networks.redundant, networks.dns1,
networks.domain_id, networks.account_id, networks.set_fields,
networks.guru_data, networks.dns2, networks.network_domain,
networks.removed, networks.created, networks.reservation_id, networks.uuid,
networks.guest_type, networks.acl_type, networks.restart_required,
networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr,
networks.display_network, networks.network_acl_id, networks.streched_l2,
networks.external_id FROM networks WHERE networks.id = 200 AND
networks.removed IS NULL
2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
connection
2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT account.id,
account.account_name, account.type, account.role_id, account.domain_id,
account.state, account.created, account.removed, account.cleanup_needed,
account.network_domain, account.uuid, account.default_zone_id,
account.default FROM account WHERE account.id = ? AND account.removed IS
NULL
### Getting Zone Information
2022-08-31 18:44:21,209 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
com.mysql.cj.jdbc.ClientPreparedStatement: SELECT data_center.id,
data_center.name, data_center.description, data_center.dns1,
data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2,
data_center.internal_dns1, data_center.internal_dns2,
data_center.router_mac_address, data_center.guest_network_cidr,
data_center.domain_id, data_center.domain, data_center.networktype,
data_center.dns_provider, data_center.dhcp_provider,
data_center.gateway_provider, data_center.vpn_provider,
data_center.userdata_provider, data_center.lb_provider,
data_center.firewall_provider, data_center.mac_address,
data_center.zone_token, data_center.removed, data_center.allocation_state,
data_center.uuid, data_center.is_security_group_enabled,
data_center.is_local_storage_enabled, data_center.sort_key FROM data_center
WHERE data_center.id = 1 AND data_center.removed IS NULL
2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
consoleproxy-1 : tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: start requested by:
-Transaction.execute:39-IpAddressManagerImpl.listAvailablePublicIps:824-IpAddressManagerImpl.fetchNewPublicIp:811-IpAddressManagerImpl.fetchNewPublicIp:803-IpAddressManagerImpl.assignPublicIpAddress:784-PublicNetworkGuru.getIp:125-PublicNetworkGuru.allocate:172-NetworkOrchestrator.allocateNic:973-NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice:823-NetworkOrchestrator$3.doInTransactionWithoutResult:797-TransactionCallbackWithExceptionNoReturn.doInTransaction:25-TransactionCallbackWithExceptionNoReturn.doInTransaction:21
2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) txn: has already been
started.
2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
connection
2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
data_center.id, data_center.name, data_center.description,
data_center.dns1, data_center.dns2, data_center.ip6_dns1,
data_center.ip6_dns2, data_center.internal_dns1, data_center.internal_dns2,
data_center.router_mac_address, data_center.guest_network_cidr,
data_center.domain_id, data_center.domain, data_center.networktype,
data_center.dns_provider, data_center.dhcp_provider,
data_center.gateway_provider, data_center.vpn_provider,
data_center.userdata_provider, data_center.lb_provider,
data_center.firewall_provider, data_center.mac_address,
data_center.zone_token, data_center.removed, data_center.allocation_state,
data_center.uuid, data_center.is_security_group_enabled,
data_center.is_local_storage_enabled, data_center.sort_key FROM data_center
WHERE data_center.id = ? AND data_center.removed IS NULL
### Account Information "system"
2022-08-31 18:44:21,211 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
com.mysql.cj.jdbc.ClientPreparedStatement: SELECT account.id,
account.account_name, account.type, account.role_id, account.domain_id,
account.state, account.created, account.removed, account.cleanup_needed,
account.network_domain, account.uuid, account.default_zone_id,
account.default FROM account WHERE account.id = 1 AND account.removed IS
NULL
2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
connection
2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT networks.id,
networks.mode, networks.broadcast_domain_type, networks.traffic_type,
networks.name, networks.display_text, networks.broadcast_uri,
networks.gateway, networks.cidr, networks.network_cidr,
networks.network_offering_id, networks.vpc_id,
networks.physical_network_id, networks.data_center_id, networks.related,
networks.guru_name, networks.state, networks.redundant, networks.dns1,
networks.domain_id, networks.account_id, networks.set_fields,
networks.guru_data, networks.dns2, networks.network_domain,
networks.removed, networks.created, networks.reservation_id, networks.uuid,
networks.guest_type, networks.acl_type, networks.restart_required,
networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr,
networks.display_network, networks.network_acl_id, networks.streched_l2,
networks.external_id FROM networks WHERE networks.id = ? AND
networks.removed IS NULL
### Getting Zone Information
2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
com.mysql.cj.jdbc.ClientPreparedStatement: SELECT data_center.id,
data_center.name, data_center.description, data_center.dns1,
data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2,
data_center.internal_dns1, data_center.internal_dns2,
data_center.router_mac_address, data_center.guest_network_cidr,
data_center.domain_id, data_center.domain, data_center.networktype,
data_center.dns_provider, data_center.dhcp_provider,
data_center.gateway_provider, data_center.vpn_provider,
data_center.userdata_provider, data_center.lb_provider,
data_center.firewall_provider, data_center.mac_address,
data_center.zone_token, data_center.removed, data_center.allocation_state,
data_center.uuid, data_center.is_security_group_enabled,
data_center.is_local_storage_enabled, data_center.sort_key FROM data_center
WHERE data_center.id = 1 AND data_center.removed IS NULL
2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
connection
2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
account_vlan_map.id, account_vlan_map.account_id,
account_vlan_map.vlan_db_id FROM account_vlan_map WHERE
account_vlan_map.account_id = ?
### Getting network information
2022-08-31 18:44:21,212 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
com.mysql.cj.jdbc.ClientPreparedStatement: SELECT networks.id,
networks.mode, networks.broadcast_domain_type, networks.traffic_type,
networks.name, networks.display_text, networks.broadcast_uri,
networks.gateway, networks.cidr, networks.network_cidr,
networks.network_offering_id, networks.vpc_id,
networks.physical_network_id, networks.data_center_id, networks.related,
networks.guru_name, networks.state, networks.redundant, networks.dns1,
networks.domain_id, networks.account_id, networks.set_fields,
networks.guru_data, networks.dns2, networks.network_domain,
networks.removed, networks.created, networks.reservation_id, networks.uuid,
networks.guest_type, networks.acl_type, networks.restart_required,
networks.specify_ip_ranges, networks.ip6_gateway, networks.ip6_cidr,
networks.display_network, networks.network_acl_id, networks.streched_l2,
networks.external_id FROM networks WHERE networks.id = null AND
networks.removed IS NULL
2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
connection
2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
domain_vlan_map.id, domain_vlan_map.domain_id, domain_vlan_map.vlan_db_id
FROM domain_vlan_map WHERE domain_vlan_map.domain_id = ?
#### Account_vlan_map - no entry for account "System" nor for Account
"Admin"
2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
com.mysql.cj.jdbc.ClientPreparedStatement: SELECT account_vlan_map.id,
account_vlan_map.account_id, account_vlan_map.vlan_db_id FROM
account_vlan_map WHERE account_vlan_map.account_id = 1
2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
connection
2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT vlan.id,
vlan.vlan_id, vlan.vlan_gateway, vlan.vlan_netmask, vlan.ip6_gateway,
vlan.ip6_cidr, vlan.data_center_id, vlan.ip4_range, vlan.ip6_range,
vlan.network_id, vlan.physical_network_id, vlan.vlan_type, vlan.uuid,
vlan.removed, vlan.created FROM vlan LEFT OUTER JOIN domain_vlan_map ON
vlan.id=domain_vlan_map.vlan_db_id LEFT OUTER JOIN account_vlan_map ON
vlan.id=account_vlan_map.vlan_db_id WHERE vlan.data_center_id = ? AND
vlan.removed IS NULL AND (domain_vlan_map.domain_id IS NULL ) AND
(account_vlan_map.account_id IS NULL )
### Checking for availeable VLANS wich are NOT assigned to ANY Account NOR
to ANY Domain --> Result is the intended vlan with expected IP Range
2022-08-31 18:44:21,213 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
com.mysql.cj.jdbc.ClientPreparedStatement: SELECT domain_vlan_map.id,
domain_vlan_map.domain_id, domain_vlan_map.vlan_db_id FROM domain_vlan_map
WHERE domain_vlan_map.domain_id = 1
2022-08-31 18:44:21,214 TRACE [c.c.u.d.GenericDaoBase]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) join search statement is
com.mysql.cj.jdbc.ClientPreparedStatement: SELECT vlan.id, vlan.vlan_id,
vlan.vlan_gateway, vlan.vlan_netmask, vlan.ip6_gateway, vlan.ip6_cidr,
vlan.data_center_id, vlan.ip4_range, vlan.ip6_range, vlan.network_id,
vlan.physical_network_id, vlan.vlan_type, vlan.uuid, vlan.removed,
vlan.created FROM vlan LEFT OUTER JOIN domain_vlan_map ON
vlan.id=domain_vlan_map.vlan_db_id
LEFT OUTER JOIN account_vlan_map ON vlan.id=account_vlan_map.vlan_db_id
WHERE vlan.data_center_id = 1 AND vlan.removed IS NULL AND
(domain_vlan_map.domain_id IS NULL ) AND (account_vlan_map.account_id IS
NULL )
2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
connection
2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
configuration.instance, configuration.component, configuration.name,
configuration.value, configuration.default_value,
configuration.description, configuration.category,
configuration.is_dynamic, configuration.scope, configuration.updated FROM
configuration WHERE configuration.name = ?
### Getting further information on the vlan --> correct choice of expected
VLAN + IP-Range
2022-08-31 18:44:21,214 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
com.mysql.cj.jdbc.ClientPreparedStatement: SELECT vlan.id, vlan.vlan_id,
vlan.vlan_gateway, vlan.vlan_netmask, vlan.ip6_gateway, vlan.ip6_cidr,
vlan.data_center_id, vlan.ip4_range, vlan.ip6_range, vlan.network_id,
vlan.physical_network_id, vlan.vlan_type, vlan.uuid, vlan.removed,
vlan.created FROM vlan LEFT OUTER JOIN domain_vlan_map ON
vlan.id=domain_vlan_map.vlan_db_id
LEFT OUTER JOIN account_vlan_map ON vlan.id=account_vlan_map.vlan_db_id
WHERE vlan.data_center_id = 1 AND vlan.removed IS NULL AND
(domain_vlan_map.domain_id IS NULL ) AND (account_vlan_map.account_id IS
NULL )
2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
connection
2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
user_ip_address.id, user_ip_address.account_id, user_ip_address.domain_id,
user_ip_address.public_ip_address, user_ip_address.data_center_id,
user_ip_address.source_nat, user_ip_address.allocated,
user_ip_address.vlan_db_id, user_ip_address.one_to_one_nat,
user_ip_address.vm_id, user_ip_address.state, user_ip_address.mac_address,
user_ip_address.source_network_id, user_ip_address.network_id,
user_ip_address.uuid, user_ip_address.physical_network_id,
user_ip_address.is_system, user_ip_address.vpc_id,
user_ip_address.dnat_vmip, user_ip_address.is_portable,
user_ip_address.display, user_ip_address.rule_state,
user_ip_address.forsystemvms, user_ip_address.removed,
user_ip_address.created FROM user_ip_address INNER JOIN vlan ON
user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id =
? AND user_ip_address.allocated IS NULL AND user_ip_address.vlan_db_id IN
(?,?,?,?) AND user_ip_address.forsystemvms = ? AND
user_ip_address.removed IS NULL AND (vlan.vlan_type = ? ) ORDER BY
user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id ASC LIMIT
0, 1 FOR UPDATE
### Check of setting system.vm.public.ip.reservation.mode.strictness -->
value "true"
2022-08-31 18:44:21,215 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
com.mysql.cj.jdbc.ClientPreparedStatement: SELECT configuration.instance,
configuration.component, configuration.name, configuration.value,
configuration.default_value, configuration.description,
configuration.category, configuration.is_dynamic, configuration.scope,
configuration.updated FROM configuration WHERE configuration.name =
x'73797374656D2E766D2E7075626C69632E69702E7265736572766174696F6E2E6D6F64652E7374726963746E657373'
2022-08-31 18:44:21,215 TRACE [c.c.u.d.GenericDaoBase]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) join search statement is
com.mysql.cj.jdbc.ClientPreparedStatement: SELECT user_ip_address.id,
user_ip_address.account_id, user_ip_address.domain_id,
user_ip_address.public_ip_address, user_ip_address.data_center_id,
user_ip_address.source_nat, user_ip_address.allocated,
user_ip_address.vlan_db_id, user_ip_address.one_to_one_nat,
user_ip_address.vm_id, user_ip_address.state, user_ip_address.mac_address,
user_ip_address.source_network_id, user_ip_address.network_id,
user_ip_address.uuid, user_ip_address.physical_network_id,
user_ip_address.is_system, user_ip_address.vpc_id,
user_ip_address.dnat_vmip, user_ip_address.is_portable,
user_ip_address.display, user_ip_address.rule_state,
user_ip_address.forsystemvms, user_ip_address.removed,
user_ip_address.created FROM user_ip_address INNER JOIN vlan ON
user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id = 1
AND user_ip_address.allocated IS NULL AND user_ip_address.vlan_db_id IN
(3,10,13,14) AND user_ip_address.forsystemvms = 1 AND
user_ip_address.removed IS NULL AND (vlan.vlan_type = 'VirtualNetwork' )
ORDER BY user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id ASC
LIMIT 0, 1 FOR UPDATE
2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
connection
2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
account_details.id, account_details.account_id, account_details.name,
account_details.value FROM account_details WHERE account_details.account_id
= ? AND account_details.name = ? ORDER BY RAND() LIMIT 1
### Trying to fetch IP-Address: No value / ip address returned - Domain
"Root" shall provide ip addresses; thats where the vlanIDs are comming from
--> user_ip_address.vlan_db_id IN (3,10,13,14)
2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
com.mysql.cj.jdbc.ClientPreparedStatement: SELECT user_ip_address.id,
user_ip_address.account_id, user_ip_address.domain_id,
user_ip_address.public_ip_address, user_ip_address.data_center_id,
user_ip_address.source_nat, user_ip_address.allocated,
user_ip_address.vlan_db_id, user_ip_address.one_to_one_nat,
user_ip_address.vm_id, user_ip_address.state, user_ip_address.mac_address,
user_ip_address.source_network_id, user_ip_address.network_id,
user_ip_address.uuid, user_ip_address.physical_network_id,
user_ip_address.is_system, user_ip_address.vpc_id,
user_ip_address.dnat_vmip, user_ip_address.is_portable,
user_ip_address.display, user_ip_address.rule_state,
user_ip_address.forsystemvms, user_ip_address.removed,
user_ip_address.created FROM user_ip_address INNER JOIN vlan ON
user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id = 1
AND user_ip_address.allocated IS NULL AND user_ip_address.vlan_db_id IN
(3,10,13,14) AND user_ip_address.forsystemvms = 1 AND
user_ip_address.removed IS NULL AND (vlan.vlan_type = 'VirtualNetwork' )
ORDER BY user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id ASC
LIMIT 0, 1 FOR UPDATE
2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
connection
2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
configuration.instance, configuration.component, configuration.name,
configuration.value, configuration.default_value,
configuration.description, configuration.category,
configuration.is_dynamic, configuration.scope, configuration.updated FROM
configuration WHERE configuration.name = ? ORDER BY RAND() LIMIT 1
### Trying to get information from the table account_detail for the account
"system". Table is empty so no result back
2022-08-31 18:44:21,216 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
com.mysql.cj.jdbc.ClientPreparedStatement: SELECT account_details.id,
account_details.account_id, account_details.name, account_details.value
FROM account_details WHERE account_details.account_id = 1 AND
account_details.name = x'7573652E73797374656D2E7075626C69632E697073' ORDER
BY RAND() LIMIT 1
2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
connection
2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
configuration.instance, configuration.component, configuration.name,
configuration.value, configuration.default_value,
configuration.description, configuration.category,
configuration.is_dynamic, configuration.scope, configuration.updated FROM
configuration WHERE configuration.name = ?
### Check for Configuration enable.account.settings.for.domain --> current
value = false
2022-08-31 18:44:21,217 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
com.mysql.cj.jdbc.ClientPreparedStatement: SELECT configuration.instance,
configuration.component, configuration.name, configuration.value,
configuration.default_value, configuration.description,
configuration.category, configuration.is_dynamic, configuration.scope,
configuration.updated FROM configuration WHERE configuration.name =
x'656E61626C652E6163636F756E742E73657474696E67732E666F722E646F6D61696E'
ORDER BY RAND() LIMIT 1
2022-08-31 18:44:21,217 WARN [c.c.n.IpAddressManagerImpl]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Unable to get ip address in
zone id=1, vlanId id=[3, 10, 13, 14]
2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Using current transaction:
consoleproxy-1 : tx-81, tx-80, tx-79, consoleproxy-1,
2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) conn: Using existing DB
connection
2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Preparing: SELECT
data_center.id, data_center.name, data_center.description,
data_center.dns1, data_center.dns2, data_center.ip6_dns1,
data_center.ip6_dns2, data_center.internal_dns1, data_center.internal_dns2,
data_center.router_mac_address, data_center.guest_network_cidr,
data_center.domain_id, data_center.domain, data_center.networktype,
data_center.dns_provider, data_center.dhcp_provider,
data_center.gateway_provider, data_center.vpn_provider,
data_center.userdata_provider, data_center.lb_provider,
data_center.firewall_provider, data_center.mac_address,
data_center.zone_token, data_center.removed, data_center.allocation_state,
data_center.uuid, data_center.is_security_group_enabled,
data_center.is_local_storage_enabled, data_center.sort_key FROM data_center
WHERE data_center.id = ? AND data_center.removed IS NULL
### Check for setting use.system.public.ips --> current value = false
2022-08-31 18:44:21,218 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
com.mysql.cj.jdbc.ClientPreparedStatement: SELECT configuration.instance,
configuration.component, configuration.name, configuration.value,
configuration.default_value, configuration.description,
configuration.category, configuration.is_dynamic, configuration.scope,
configuration.updated FROM configuration WHERE configuration.name =
x'7573652E73797374656D2E7075626C69632E697073'
### getting information for the current data_center / zone
2022-08-31 18:44:21,219 TRACE [c.c.u.d.T.Statement]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing:
com.mysql.cj.jdbc.ClientPreparedStatement: SELECT data_center.id,
data_center.name, data_center.description, data_center.dns1,
data_center.dns2, data_center.ip6_dns1, data_center.ip6_dns2,
data_center.internal_dns1, data_center.internal_dns2,
data_center.router_mac_address, data_center.guest_network_cidr,
data_center.domain_id, data_center.domain, data_center.networktype,
data_center.dns_provider, data_center.dhcp_provider,
data_center.gateway_provider, data_center.vpn_provider,
data_center.userdata_provider, data_center.lb_provider,
data_center.firewall_provider, data_center.mac_address,
data_center.zone_token, data_center.removed, data_center.allocation_state,
data_center.uuid, data_center.is_security_group_enabled,
data_center.is_local_storage_enabled, data_center.sort_key FROM data_center
WHERE data_center.id = 1 AND data_center.removed IS NULL
2022-08-31 18:44:21,219 DEBUG [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Rolling back the
transaction: Time = 17 Name = consoleproxy-1; called by
-TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-Transaction.execute:38-IpAddressManagerImpl.listAvailablePublicIps:824-IpAddressManagerImpl.fetchNewPublicIp:811-IpAddressManagerImpl.fetchNewPublicIp:803-IpAddressManagerImpl.assignPublicIpAddress:784-PublicNetworkGuru.getIp:125-PublicNetworkGuru.allocate:172-NetworkOrchestrator.allocateNic:973-NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice:823
2022-08-31 18:44:21,219 TRACE [c.c.u.d.T.Lock]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) SQL
com.mysql.cj.jdbc.ClientPreparedStatement: SELECT user_ip_address.id,
user_ip_address.account_id, user_ip_address.domain_id,
user_ip_address.public_ip_address, user_ip_address.data_center_id,
user_ip_address.source_nat, user_ip_address.allocated,
user_ip_address.vlan_db_id, user_ip_address.one_to_one_nat,
user_ip_address.vm_id, user_ip_address.state, user_ip_address.mac_address,
user_ip_address.source_network_id, user_ip_address.network_id,
user_ip_address.uuid, user_ip_address.physical_network_id,
user_ip_address.is_system, user_ip_address.vpc_id,
user_ip_address.dnat_vmip, user_ip_address.is_portable,
user_ip_address.display, user_ip_address.rule_state,
user_ip_address.forsystemvms, user_ip_address.removed,
user_ip_address.created FROM user_ip_address INNER JOIN vlan ON
user_ip_address.vlan_db_id=vlan.id WHERE user_ip_address.data_center_id = 1
AND user_ip_address.allocated IS NULL AND user_ip_address.vlan_db_id IN
(3,10,13,14) AND user_ip_address.forsystemvms = 1 AND
user_ip_address.removed IS NULL AND (vlan.vlan_type = 'VirtualNetwork' )
ORDER BY user_ip_address.forsystemvms DESC , user_ip_address.vlan_db_id ASC
LIMIT 0, 1 FOR UPDATE took 4
2022-08-31 18:44:21,219 TRACE [c.c.u.d.T.Connection]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Closing DB connection:
dbconn623913035
2022-08-31 18:44:21,220 WARN [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-a0aef6e9) (logid:d5e7c7c1) Unable to allocate proxy
[Console VM instance {id: "86", name: "v-86-VM", uuid:
"6f77acb4-c65e-49e8-ae03-1175b3974412", type="ConsoleProxy"}] on zone [1]
due to [Insufficient address capacity].
com.cloud.exception.InsufficientAddressCapacityException: Insufficient
address capacityScope=interface com.cloud.dc.DataCenter; id=1
following the settings of our global config which are mentioned in the
above log:
*system.vm.public.ip.reservation.mode.strictness* - true
*use.system.public.ips* - true
*enable.account.settings.for.domain* - false
The challange now is to get all the things together into a context...
From my impression changeing the value for 'use.system.public.ips' may not
be helping, as at this point the "warning" already has been dropped. But
would only be a "workaround".
Questions from my side:
- Looking into the web GUI for the IP-Ranges for the public network i am
getting quite curios. The current entry for '[Root] system' is strange.
Clicking on it i am getting a view stateing that this range shall be
"connected" to the Account 'System', the Domain 'Root' and SystemVMs 'true'.
While the 'System VMs = true' value can be found in the database, the vlan
/ ip - range isn't connected to a account or domain at all in the database.
- What is the overall 'effect' of setting
*system.vm.public.ip.reservation.mode.strictness?
*From the description for me a "true" will prevent the usage of of public
IPs from the system pool. But what is defined as "system Pool"? Looking at
the GUI I would assume that the "account" System has a dedicated pool of
public IPs for systemVMs (speaking that this would fail as there is no
reference in the DB for a mapping of any Pool / Range for the account
'system').
To summerize this seems more like a "unlucky" configuration or missing
db.entry / bug throughout versions. Or a mix of both.
How ever i don't really understand why there is a query to find VLANs not
attached to a Account or an Domain - or is this the SystemPool?
Thanks again for all the help!
Chris
Am Mi., 31. Aug. 2022 um 10:52 Uhr schrieb Slavka Peleva <
slavkap@storpool.com>:
> Hi Chris,
>
> According to the logs you shared and looking at the code, I guess CS gets
> the VLAN IDs by domain id, not by zone id. But to be more precise in this,
> can you enable the TRACE level in your logs and share the exact query that
> CS executes?
> If I'm right with this, you can enable *just for now* the setting
> `use.system.public.ips` (To redeploy the system VMs). This should pick
> VLANs by zone ID. I hope that the deployment won't fail this time! If the
> system VMs are recreated, can you disable the setting
> `use.system.public.ips` (because I don't know if this will affect something
> else)?
>
> The discussion started about a problem migrating the system VMs, and
> probably many other issues will appear.
> Can you tell from which version you started the upgrade? (two weeks ago,
> I upgraded one of my environments from 4.11.3.0 to the main branch
> 4.18.0.0-SNAPSHOT without any issues)
> Did the upgrade was successful for the management/agents?
> Did you restart the agents on all machines? My guess is the new code
> isn't loaded. It's strange that CS is looking for systemvm.iso when 4.17
> doesn't know about it.
> Can you share the content in the directory
> `/usr/share/cloudstack-agent/lib/` from the hypervisor where the migration
> failed?
>
> Best regards,
> Slavka
>
>
>
> On Wed, Aug 31, 2022 at 2:35 AM Vash_X@gmx.de <Va...@gmx.de> wrote:
>
>> Hi Nicolas,
>>
>> > What is the value for the setting
>> ‘system.vm.public.ip.reservation.mode.strictness’?
>> This is set to 'true'.
>>
>> > Can you please check if this query gets any result?
>> Yes the query gives me results - all availeable "public" ip addresses of
>> the reserved range for the systemVMs
>>
>> # public_ip_address vlan_db_id vlan_id
>> 1 10.129.0.10 9 1000
>> 2 10.129.0.11 9 1000
>> 3 10.129.0.12 9 1000
>> 4 10.129.0.13 9 1000
>> 5 10.129.0.14 9 1000
>> 6 10.129.0.15 9 1000
>> 7 10.129.0.16 9 1000
>> 8 10.129.0.18 9 1000
>> 9 10.129.0.19 9 1000
>> Regarding the vlanId id=[3, 10, 13, 14]
>> I looked up my table "vlan" and am quite shocked:
>>
>> the ids 3, 13, 14 have a value for "range removed", so i don't know why
>> the
>> console proxy is looking for these networks.
>> the id "10" is is still active - however the ipv4 range is not "the right
>> one". That is assigned as the public ip-range for the root-account but not
>> for use of systemvms...
>>
>> Am Di., 30. Aug. 2022 um 16:43 Uhr schrieb Nicolas Vazquez <
>> Nicolas.Vazquez@shapeblue.com>:
>>
>> > Hi Chris,
>> >
>> > What is the value for the setting
>> > ‘system.vm.public.ip.reservation.mode.strictness’?
>> >
>> > Looks like the error is not being able to get an IP address for zone
>> id=1,
>> > vlanId id=[3, 10, 13, 14].
>> > If I understood correctly you have defined a range for system VMs? Can
>> you
>> > please check if this query gets any result?
>> >
>> > select i.public_ip_address, i.vlan_db_id, v.vlan_id from
>> user_ip_address i
>> > join vlan v on (i.vlan_db_id = v.id)
>> > where i.forsystemvms = 1 and i.allocated is null and i.state = 'Free'
>> and
>> > v.removed is null and i.data_center_id = 1;
>> >
>> > Regards,
>> > Nicolas Vazquez
>> >
>> >
>> > From: Vash_X@gmx.de <Va...@gmx.de>
>> > Date: Tuesday, 30 August 2022 at 09:38
>> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>, Slavka
>> > Peleva <sl...@storpool.com>
>> > Subject: Re: Error migrate System VMs after upgrageing to 4.17
>> > Hi Slavka,
>> >
>> > that was my first thought, too. I checked beforehand and confirmed with
>> > your provided SQL Statements:
>> > *`select * from user_ip_address where forsystemvms = 1;`* <---- returns
>> my
>> > defined (Public) IPs out of the correct range for SystemVMs
>> > *`SELECT * FROM clou*d.user_ip_address where allocated IS NULL AND
>> removed
>> > IS NULL; <-- All IPs shown beside the one which is currently in use for
>> the
>> > running ssvm. Meaning from my point of view "free" for allocation for
>> other
>> > system VMs.
>> >
>> > Querys performed directly on the DB-Server.
>> > I can't see a reason why there shouldn't be any IPs left for allocation.
>> >
>> > Also it is interesting why the "reasoning" for the failing deployment
>> > changes from "no IP availeable" to "no useable volumes found".
>> > How ever i can't see anything missing at the moment, what makes me even
>> > more puzzeled.
>> >
>> > Regards
>> > Chris
>> >
>> > Am Di., 30. Aug. 2022 um 13:31 Uhr schrieb Slavka Peleva
>> > <sl...@storpool.com.invalid>:
>> >
>> > > Hi Chris,
>> > >
>> > > The deployment fails because CS cannot find a free IP address to
>> allocate
>> > > for the system VM. Can you check if you explicitly added a range for
>> the
>> > > system VMs, or you can check with this `select * from user_ip_address
>> > where
>> > > forsystemvms = 1;` If you don't have any ranges, you can get free IPs
>> > > with `SELECT
>> > > * FROM cloud.user_ip_address where allocated IS NULL AND removed IS
>> > NULL;`
>> > > . To get the exact query executed by CS, you need to increase the log
>> > level
>> > > to TRACE.
>> > >
>> > > Best regards,
>> > > Slavka
>> > >
>> >
>> >
>> >
>> > > On Mon, Aug 29, 2022 at 8:03 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
>> > >
>> > > > Hi Slavka,
>> > > >
>> > > > no, i am not using any kind of custome offerings.
>> > > >
>> > > > I try to provide a more detailled log this time.
>> > > > Situation:
>> > > > v-83-VM was the "broken" consoleproxy. So the following log starts
>> with
>> > > > it's expunging.
>> > > > Afterwards the new consoleproxy shall be deployed - v-85-VM.
>> > > >
>> > > > This first deployment (logid:dcfe0999) fails due to Insufficent
>> address
>> > > > capacity - which i can't really understand or see in the database,
>> > > checking
>> > > > the availeable IPs for systemVMs (given that i might looking up the
>> > wrong
>> > > > tables....).
>> > > > How ever CS is working as expected trying to deploy the console
>> proxy
>> > > > (logid:440fd101) Sync job-4288).
>> > > > That job fails with "Unable to create deployment, no usable volumes
>> > found
>> > > > for the VM: 85".
>> > > > Afterwards i have another try to start the consoleproxy (job-4289)
>> > which
>> > > > fails for the same reason. And thats how things go on in the log
>> > file...
>> > > >
>> > > >
>> > > > 2022-08-29 17:01:16,224 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>> > (logid:5fe8559a)
>> > > > Cleaning up hypervisor data structures (ex. SRs in XenServer) for
>> > managed
>> > > > storage. Data from VM instance {id: "83", name: "v-83-VM", uuid:
>> > > > "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}.
>> > > > 2022-08-29 17:01:16,225 DEBUG [o.a.c.e.o.VolumeOrchestrator]
>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>> > (logid:5fe8559a)
>> > > > Cleaning storage for vm: 83
>> > > > 2022-08-29 17:01:16,233 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>> > (logid:5fe8559a)
>> > > > Expunged VM instance {id: "83", name: "v-83-VM", uuid:
>> > > > "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
>> > > > 2022-08-29 17:01:16,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>> > (logid:5fe8559a)
>> > > > Complete async job-4286, jobStatus: SUCCEEDED, resultCode: 0,
>> result:
>> > > >
>> > >
>> >
>> org.apache.cloudstack.api.response.SystemVmResponse/systemvm/{"id":"b1d1f9c6-eafe-4527-9335-e649c646aab0","systemvmtype":"consoleproxy","zoneid":"48e2e928-3300-43b5-8e3a-d9b358022502","zonename":"xxxxxxxxxx","dns1":"10.129.0.4","dns2":"195.10.195.195","name":"v-83-VM","templateid":"c20799e4-a990-449a-aae6-4b67374c7aff","templatename":"systemvm-kvm-4.17.0","created":"2022-08-29T10:38:26+0200","state":"Stopped","isdynamicallyscalable":"false","hasannotations":"false"}
>> > > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>> > (logid:5fe8559a)
>> > > > Publish async job-4286 complete on message bus
>> > > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>> > (logid:5fe8559a)
>> > > > Wake up jobs related to job-4286
>> > > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>> > (logid:5fe8559a)
>> > > > Update db status for job-4286
>> > > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
>> > (logid:5fe8559a)
>> > > > Wake up jobs joined with job-4286 and disjoin all subjobs created
>> from
>> > > job-
>> > > > 4286
>> > > > 2022-08-29 17:01:16,260 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286) (logid:5fe8559a) Done
>> > > executing
>> > > > org.apache.cloudstack.api.command.admin.systemvm.DestroySystemVmCmd
>> for
>> > > > job-4286
>> > > > 2022-08-29 17:01:16,260 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286) (logid:5fe8559a) Remove
>> > > > job-4286 from job monitoring
>> > > > 2022-08-29 17:01:17,691 DEBUG [o.a.c.h.HAManagerImpl]
>> > > > (BackgroundTaskPollManager-6:ctx-6af18142) (logid:fe263a70) HA
>> health
>> > > check
>> > > > task is running...
>> > > >
>> > > > 2022-08-29 17:01:21,694 DEBUG [o.a.c.h.HAManagerImpl]
>> > > > (BackgroundTaskPollManager-6:ctx-89b1442f) (logid:0efc40b2) HA
>> health
>> > > check
>> > > > task is running...
>> > > > 2022-08-29 17:01:25,698 DEBUG [o.a.c.h.HAManagerImpl]
>> > > > (BackgroundTaskPollManager-5:ctx-62313d05) (logid:7073663f) HA
>> health
>> > > check
>> > > > task is running...
>> > > > 2022-08-29 17:01:25,767 DEBUG [c.c.s.StatsCollector]
>> > > > (secstorage-1:ctx-c80e5a6e) (logid:f8e3eb60) Verifying image storage
>> > [1].
>> > > > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
>> > > > 2022-08-29 17:01:25,773 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
>> > > > (secstorage-1:ctx-c80e5a6e) (logid:f8e3eb60) Zone [1] is ready to
>> > launch
>> > > > secondary storage VM.
>> > > > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Zone 1 is ready to
>> > launch
>> > > > console proxy
>> > > > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Expand console proxy
>> > > standby
>> > > > capacity for zone xxxxxxxxxx
>> > > > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating console
>> proxy
>> > > > standby capacity for zone [1].
>> > > > 2022-08-29 17:01:26,032 INFO [c.c.c.ConsoleProxyManagerImpl]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) No stopped console
>> proxy
>> > > is
>> > > > available, need to allocate a new console proxy
>> > > > 2022-08-29 17:01:26,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Assign console proxy
>> > from
>> > > a
>> > > > newly started instance for request from data center : 1
>> > > > 2022-08-29 17:01:26,056 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing
>> network
>> > > > configuration for offering [Network Offering
>> > > > [1-Public-System-Public-Network]: Network {"id": 200, "name":
>> "null",
>> > > > "uuid": "1b0bf965-a391-478d-833d-3e37e4a15eb8",
>> "networkofferingid": 1}
>> > > > 2022-08-29 17:01:26,056 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
>> > > > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account {"id":
>> 1,
>> > > > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
>> > > > 2022-08-29 17:01:26,064 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing
>> network
>> > > > configuration for offering [Network Offering
>> > > > [3-Control-System-Control-Network]: Network {"id": 202, "name":
>> "null",
>> > > > "uuid": "2af42518-fbf4-4a35-9d6b-370e6ad315c7",
>> "networkofferingid": 3}
>> > > > 2022-08-29 17:01:26,064 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
>> > > > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account {"id":
>> 1,
>> > > > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
>> > > > 2022-08-29 17:01:26,069 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing
>> network
>> > > > configuration for offering [Network Offering
>> > > > [2-Management-System-Management-Network]: Network {"id": 201,
>> "name":
>> > > > "null", "uuid": "53711d87-a3c6-4f33-bfce-c30e0de72760",
>> > > > "networkofferingid": 2}
>> > > > 2022-08-29 17:01:26,069 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
>> > > > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account {"id":
>> 1,
>> > > > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
>> > > > 2022-08-29 17:01:26,077 INFO [c.c.v.VirtualMachineManagerImpl]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) allocating virtual
>> > machine
>> > > > from template:c20799e4-a990-449a-aae6-4b67374c7aff with
>> > hostname:v-85-VM
>> > > > and 3 networks
>> > > > 2022-08-29 17:01:26,079 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating entries
>> for
>> > VM:
>> > > > VM instance {id: "85", name: "v-85-VM", uuid:
>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
>> > > > 2022-08-29 17:01:26,082 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating nics for
>> VM
>> > > > instance {id: "85", name: "v-85-VM", uuid:
>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
>> > > > 2022-08-29 17:01:26,083 DEBUG [o.a.c.e.o.NetworkOrchestrator]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating nic for
>> vm VM
>> > > > instance {id: "85", name: "v-85-VM", uuid:
>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} in
>> network
>> > > > Network {"id": 200, "name": "null", "uuid":
>> > > > "1b0bf965-a391-478d-833d-3e37e4a15eb8", "networkofferingid": 1} with
>> > > > requested profile NicProfile
>> > > >
>> > >
>> >
>> {"broadcastUri":null,"iPv4Address":null,"id":0,"reservationId":null,"vmId":0}
>> > > > 2022-08-29 17:01:26,089 WARN [c.c.n.IpAddressManagerImpl]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to get ip
>> address
>> > > in
>> > > > zone id=1, vlanId id=[3, 10, 13, 14]
>> > > > 2022-08-29 17:01:26,090 DEBUG [c.c.u.d.T.Transaction]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Rolling back the
>> > > > transaction: Time = 8 Name = consoleproxy-1; called by
>> > > >
>> > >
>> >
>> -TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-Transaction.execute:38-IpAddressManagerImpl.listAvailablePublicIps:824-IpAddressManagerImpl.fetchNewPublicIp:811-IpAddressManagerImpl.fetchNewPublicIp:803-IpAddressManagerImpl.assignPublicIpAddress:784-PublicNetworkGuru.getIp:125-PublicNetworkGuru.allocate:172-NetworkOrchestrator.allocateNic:973-NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice:823
>> > > > 2022-08-29 17:01:26,090 WARN [c.c.c.ConsoleProxyManagerImpl]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to allocate
>> proxy
>> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
>> zone
>> > [1]
>> > > > due to [Insufficient address capacity].
>> > > > com.cloud.exception.InsufficientAddressCapacityException:
>> Insufficient
>> > > > address capacityScope=interface com.cloud.dc.DataCenter; id=1
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
>> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.network.IpAddressManagerImpl.listAvailablePublicIps(IpAddressManagerImpl.java:824)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl.java:811)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl.java:803)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.network.IpAddressManagerImpl.assignPublicIpAddress(IpAddressManagerImpl.java:784)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.network.guru.PublicNetworkGuru.getIp(PublicNetworkGuru.java:125)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.network.guru.PublicNetworkGuru.allocate(PublicNetworkGuru.java:172)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocateNic(NetworkOrchestrator.java:973)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice(NetworkOrchestrator.java:823)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$3.doInTransactionWithoutResult(NetworkOrchestrator.java:797)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
>> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocate(NetworkOrchestrator.java:778)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.vm.VirtualMachineManagerImpl$1.doInTransactionWithoutResult(VirtualMachineManagerImpl.java:478)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
>> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:469)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:528)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:700)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
>> > > > at
>> > >
>> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
>> > > > at
>> > > >
>> >
>> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>> > > > 2022-08-29 17:01:26,091 WARN [c.c.c.ConsoleProxyManagerImpl]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to allocate
>> > console
>> > > > proxy standby capacity for zone [1] due to [Unable to allocate proxy
>> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
>> zone
>> > [1]
>> > > > due to [Insufficient address capacity].].
>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to allocate
>> > proxy
>> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
>> zone
>> > [1]
>> > > > due to [Insufficient address capacity].
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:704)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
>> > > > at
>> > >
>> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
>> > > > at
>> > > >
>> >
>> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>> > > > Caused by: com.cloud.exception.InsufficientAddressCapacityException:
>> > > > Insufficient address capacityScope=interface
>> com.cloud.dc.DataCenter;
>> > > id=1
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
>> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
>> > > > ... 38 more
>> > > > 2022-08-29 17:01:26,091 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) received console
>> proxy
>> > > alert
>> > > > 2022-08-29 17:01:26,092 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Console proxy
>> creation
>> > > > failure, zone: xxxxxxxxxx
>> > > > 2022-08-29 17:01:26,093 WARN [c.c.a.AlertManagerImpl]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) alertType=[10]
>> > > > dataCenterId=[1] podId=[null] clusterId=[null] message=[Console
>> proxy
>> > > > creation failure. zone: xxxxxxxxxx, error details: Unable to
>> allocate
>> > > proxy
>> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
>> zone
>> > [1]
>> > > > due to [Insufficient address capacity].].
>> > > > 2022-08-29 17:01:26,096 WARN [c.c.a.AlertManagerImpl]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) No recipients set in
>> > > global
>> > > > setting 'alert.email.addresses', skipping sending alert with subject
>> > > > [Console proxy creation failure. zone: xxxxxxxxxx, error details:
>> > Unable
>> > > to
>> > > > allocate proxy [Console VM instance {id: "85", name: "v-85-VM",
>> uuid:
>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
>> zone
>> > [1]
>> > > > due to [Insufficient address capacity].] and content [Console proxy
>> > > > creation failure (zone xxxxxxxxxx)].
>> > > > 2022-08-29 17:01:26,096 WARN [c.c.v.SystemVmLoadScanner]
>> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unexpected exception
>> > > Unable
>> > > > to allocate proxy [Console VM instance {id: "85", name: "v-85-VM",
>> > uuid:
>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
>> zone
>> > [1]
>> > > > due to [Insufficient address capacity].
>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to allocate
>> > proxy
>> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on
>> zone
>> > [1]
>> > > > due to [Insufficient address capacity].
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:704)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
>> > > > at
>> > >
>> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
>> > > > at
>> > > >
>> >
>> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>> > > > Caused by: com.cloud.exception.InsufficientAddressCapacityException:
>> > > > Insufficient address capacityScope=interface
>> com.cloud.dc.DataCenter;
>> > > id=1
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
>> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
>> > > > ... 38 more
>> > > > 2022-08-29 17:01:29,700 DEBUG [o.a.c.h.HAManagerImpl]
>> > > > (BackgroundTaskPollManager-5:ctx-994465bd) (logid:1e2822fd) HA
>> health
>> > > check
>> > > > task is running...
>> > > > 2022-08-29 17:01:30,444 DEBUG
>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
>> > > > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 2
>> routers
>> > to
>> > > > update status.
>> > > > 2022-08-29 17:01:30,449 DEBUG
>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
>> > > > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 1 VPC's
>> to
>> > > > update Redundant State.
>> > > > 2022-08-29 17:01:30,451 DEBUG
>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
>> > > > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 0
>> networks
>> > to
>> > > > update RvR status.
>> > > > 2022-08-29 17:01:30,459 DEBUG [c.c.a.t.Request]
>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
>> > > > 6-4833769775052102463: Sending { Cmd , MgmtId: 90520740759984, via:
>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
>> > > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
>> > > router.name
>> > >
>> >
>> ":"r-79-VM","router.ip":"169.254.144.198"},"wait":"30","bypassHostMaintenance":"false"}}]
>> > > > }
>> > > > 2022-08-29 17:01:30,643 DEBUG [c.c.a.t.Request]
>> > > > (AgentManager-Handler-12:null) (logid:) Seq 6-4833769775052102463:
>> > > > Processing: { Ans: , MgmtId: 90520740759984, via: 6, Ver: v1,
>> Flags:
>> > 10,
>> > > >
>> > >
>> >
>> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"PRIMARY","result":"true","details":"Status:
>> > > > PRIMARY
>> > > > ","wait":"0","bypassHostMaintenance":"false"}}] }
>> > > > 2022-08-29 17:01:30,643 DEBUG [c.c.a.t.Request]
>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
>> > > > 6-4833769775052102463: Received: { Ans: , MgmtId: 90520740759984,
>> via:
>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer
>> } }
>> > > > 2022-08-29 17:01:30,643 DEBUG [c.c.a.m.AgentManagerImpl]
>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08)
>> Details
>> > > from
>> > > > executing class com.cloud.agent.api.CheckRouterCommand: Status:
>> PRIMARY
>> > > >
>> > > > 2022-08-29 17:01:30,655 DEBUG [c.c.a.t.Request]
>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
>> > > > 1-5410793477308947401: Sending { Cmd , MgmtId: 90520740759984, via:
>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
>> > > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
>> > > router.name
>> > >
>> >
>> ":"r-80-VM","router.ip":"169.254.217.200"},"wait":"30","bypassHostMaintenance":"false"}}]
>> > > > }
>> > > > 2022-08-29 17:01:30,796 DEBUG [c.c.a.t.Request]
>> > > > (AgentManager-Handler-7:null) (logid:) Seq 1-5410793477308947401:
>> > > > Processing: { Ans: , MgmtId: 90520740759984, via: 1, Ver: v1,
>> Flags:
>> > 10,
>> > > >
>> > >
>> >
>> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"BACKUP","result":"true","details":"Status:
>> > > > BACKUP
>> > > > ","wait":"0","bypassHostMaintenance":"false"}}] }
>> > > > 2022-08-29 17:01:30,796 DEBUG [c.c.a.t.Request]
>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
>> > > > 1-5410793477308947401: Received: { Ans: , MgmtId: 90520740759984,
>> via:
>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer
>> } }
>> > > > 2022-08-29 17:01:30,796 DEBUG [c.c.a.m.AgentManagerImpl]
>> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08)
>> Details
>> > > from
>> > > > executing class com.cloud.agent.api.CheckRouterCommand: Status:
>> BACKUP
>> > > >
>> > > > 2022-08-29 17:01:33,703 DEBUG [o.a.c.h.HAManagerImpl]
>> > > > (BackgroundTaskPollManager-5:ctx-70bca202) (logid:b332dbe0) HA
>> health
>> > > check
>> > > > task is running...
>> > > > 2022-08-29 17:01:34,083 DEBUG [c.c.s.StatsCollector]
>> > > > (StatsCollector-6:ctx-213e2ecc) (logid:aa0584c9) AutoScaling
>> Monitor is
>> > > > running...
>> > > > 2022-08-29 17:01:37,707 DEBUG [o.a.c.h.HAManagerImpl]
>> > > > (BackgroundTaskPollManager-6:ctx-57493517) (logid:5093e75f) HA
>> health
>> > > check
>> > > > task is running...
>> > > > 2022-08-29 17:01:38,640 DEBUG [c.c.s.StatsCollector]
>> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) HostStatsCollector
>> is
>> > > > running...
>> > > > 2022-08-29 17:01:38,655 DEBUG [c.c.a.t.Request]
>> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
>> > > 1-5410793477308947402:
>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>> GetHostStatsAnswer }
>> > }
>> > > > 2022-08-29 17:01:38,707 DEBUG [c.c.a.t.Request]
>> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
>> > > 5-8959067033723473395:
>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>> > > > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>> GetHostStatsAnswer }
>> > }
>> > > > 2022-08-29 17:01:38,717 DEBUG [c.c.a.t.Request]
>> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
>> > > 6-4833769775052102464:
>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>> GetHostStatsAnswer }
>> > }
>> > > > 2022-08-29 17:01:38,725 DEBUG [c.c.a.t.Request]
>> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
>> > > 7-295548725546192472:
>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>> > > > 7(srv-5-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>> GetHostStatsAnswer }
>> > }
>> > > > 2022-08-29 17:01:41,709 DEBUG [o.a.c.h.HAManagerImpl]
>> > > > (BackgroundTaskPollManager-5:ctx-a65ae462) (logid:bc2b7320) HA
>> health
>> > > check
>> > > > task is running...
>> > > > 2022-08-29 17:01:45,712 DEBUG [o.a.c.h.HAManagerImpl]
>> > > > (BackgroundTaskPollManager-2:ctx-05491ace) (logid:b2232c4d) HA
>> health
>> > > check
>> > > > task is running...
>> > > > 2022-08-29 17:01:47,019 DEBUG [c.c.a.m.AgentManagerImpl]
>> > > > (AgentManager-Handler-14:null) (logid:) Ping from Routing host
>> > > > 7(srv-5-comp.aaaaaaaa.bbb)
>> > > >
>> > > > 2022-08-29 17:01:49,715 DEBUG [o.a.c.h.HAManagerImpl]
>> > > > (BackgroundTaskPollManager-2:ctx-7943581c) (logid:504c98ff) HA
>> health
>> > > check
>> > > > task is running...
>> > > > 2022-08-29 17:01:51,881 DEBUG [c.c.a.m.AgentManagerImpl]
>> > > > (AgentManager-Handler-13:null) (logid:) Ping from Routing host
>> > > > 1(srv-2-comp.aaaaaaaa.bbb)
>> > > >
>> > > > 2022-08-29 17:01:53,718 DEBUG [o.a.c.h.HAManagerImpl]
>> > > > (BackgroundTaskPollManager-4:ctx-b11cfa22) (logid:3163613f) HA
>> health
>> > > check
>> > > > task is running...
>> > > > 2022-08-29 17:01:53,955 DEBUG [c.c.a.t.Request]
>> > > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
>> > > 1-5410793477308947403:
>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer
>> } }
>> > > > 2022-08-29 17:01:54,025 DEBUG [c.c.a.t.Request]
>> > > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
>> > > 5-8959067033723473396:
>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>> > > > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer
>> } }
>> > > > 2022-08-29 17:01:54,096 DEBUG [c.c.a.t.Request]
>> > > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
>> > > 6-4833769775052102465:
>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer
>> } }
>> > > > 2022-08-29 17:01:54,217 DEBUG [c.c.a.t.Request]
>> > > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
>> > > 7-295548725546192473:
>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>> > > > 7(srv-5-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer
>> } }
>> > > > 2022-08-29 17:01:55,765 DEBUG [c.c.s.StatsCollector]
>> > > > (secstorage-1:ctx-7d9995d7) (logid:e388a8b8) Verifying image storage
>> > [1].
>> > > > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
>> > > > 2022-08-29 17:01:55,771 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
>> > > > (secstorage-1:ctx-7d9995d7) (logid:e388a8b8) Zone [1] is ready to
>> > launch
>> > > > secondary storage VM.
>> > > > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Zone 1 is ready to
>> > launch
>> > > > console proxy
>> > > > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Expand console proxy
>> > > standby
>> > > > capacity for zone xxxxxxxxxx
>> > > > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Allocating console
>> proxy
>> > > > standby capacity for zone [1].
>> > > > 2022-08-29 17:01:56,034 INFO [c.c.c.ConsoleProxyManagerImpl]
>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Found a stopped
>> console
>> > > > proxy, starting it. Vm id : 85
>> > > > 2022-08-29 17:01:56,035 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) start parameter
>> value of
>> > > > enterHardwareSetup == <very null> during processing of queued job
>> > > > 2022-08-29 17:01:56,044 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Sync job-4288
>> execution
>> > on
>> > > > object VmWorkJobQueue.85
>> > > > 2022-08-29 17:01:56,173 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (AsyncJobMgr-Heartbeat-1:ctx-e75826b1) (logid:e80fe659) Execute
>> > > sync-queue
>> > > > item: SyncQueueItemVO {id:2105, queueId: 2105, contentType:
>> AsyncJob,
>> > > > contentId: 4288, lastProcessMsid: 90520740759984,
>> lastprocessNumber: 1,
>> > > > lastProcessTime: Mon Aug 29 17:01:56 CEST 2022, created: Mon Aug 29
>> > > > 17:01:56 CEST 2022}
>> > > > 2022-08-29 17:01:56,174 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (AsyncJobMgr-Heartbeat-1:ctx-e75826b1) (logid:e80fe659) Schedule
>> queued
>> > > > job-4288
>> > > > 2022-08-29 17:01:56,179 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
>> (logid:7e671dd0)
>> > > Add
>> > > > job-4288 into job monitoring
>> > > > 2022-08-29 17:01:56,186 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
>> (logid:ba10fec5)
>> > > > Executing AsyncJobVO: {id:4288, userId: 1, accountId: 1,
>> instanceType:
>> > > > null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
>> > > >
>> > >
>> >
>> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAVXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
>> > > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>> > > > result: null, initMsid: 90520740759984, completeMsid: null,
>> > lastUpdated:
>> > > > null, lastPolled: null, created: Mon Aug 29 17:01:56 CEST 2022,
>> > removed:
>> > > > null}
>> > > > 2022-08-29 17:01:56,186 DEBUG [c.c.v.VmWorkJobDispatcher]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
>> (logid:ba10fec5)
>> > > Run
>> > > > VM work job: com.cloud.vm.VmWorkStart for VM 85, job origin: 3445
>> > > > 2022-08-29 17:01:56,187 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>> > > > (logid:ba10fec5) Execute VM work job:
>> > > >
>> > >
>> >
>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
>> > > > 2022-08-29 17:01:56,189 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>> > > > (logid:ba10fec5) orchestrating VM start for 'v-85-VM'
>> > > > com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
>> > > > 2022-08-29 17:01:56,196 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>> > > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
>> > > > transited from [Stopped] to [Starting] with event [StartRequested].
>> > VM's
>> > > > original host: null, new host: null, host before state transition:
>> null
>> > > > 2022-08-29 17:01:56,196 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>> > > > (logid:ba10fec5) Successfully transitioned to start state for VM
>> > instance
>> > > > {id: "85", name: "v-85-VM", uuid:
>> > "110d8620-c029-4ca9-9084-210099725a80",
>> > > > type="ConsoleProxy"} reservation id =
>> > > 0ab68202-d8be-469c-9b2a-87a55c2dc489
>> > > > 2022-08-29 17:01:56,200 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>> > > > (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts:
>> null
>> > > > 2022-08-29 17:01:56,202 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>> > > > (logid:ba10fec5) VM start attempt #1
>> > > > 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>> > > > (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
>> > > > 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>> > > > (logid:ba10fec5) Trying to allocate a host and storage pools from
>> dc:1,
>> > > > pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
>> > > > 1073741824
>> > > > 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>> > > > (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
>> > > > 2022-08-29 17:01:56,212 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>> > > > (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
>> > > > 2022-08-29 17:01:56,214 DEBUG [c.c.d.FirstFitPlanner]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>> > > > (logid:ba10fec5) Searching all possible resources under this Zone: 1
>> > > > 2022-08-29 17:01:56,215 DEBUG [c.c.d.FirstFitPlanner]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>> > > > (logid:ba10fec5) Listing clusters in order of aggregate capacity,
>> that
>> > > have
>> > > > (at least one host with) enough CPU and RAM capacity under this
>> Zone: 1
>> > > > 2022-08-29 17:01:56,218 DEBUG [c.c.d.FirstFitPlanner]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>> > > > (logid:ba10fec5) Removing from the clusterId list these clusters
>> from
>> > > avoid
>> > > > set: []
>> > > > 2022-08-29 17:01:56,224 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>> > > > (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
>> > > > 2022-08-29 17:01:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested
>> with
>> > > > Custom[UEFI] Boot Type false
>> > > > 2022-08-29 17:01:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in dc:
>> 1
>> > > > pod:1 cluster:1
>> > > > 2022-08-29 17:01:56,226 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4
>> > hosts
>> > > to
>> > > > check for allocation: [Host {"id": "5", "name":
>> > > "srv-4-comp.aaaaaaaa.bbb",
>> > > > "uuid": "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"},
>> Host
>> > > > {"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb", "uuid":
>> > > > "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
>> {"id":
>> > > "6",
>> > > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
>> > > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
>> {"id":
>> > > "1",
>> > > > "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
>> > > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
>> > > > 2022-08-29 17:01:56,232 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for
>> allocation
>> > > > after prioritization: [Host {"id": "5", "name":
>> > > "srv-4-comp.aaaaaaaa.bbb",
>> > > > "uuid": "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"},
>> Host
>> > > > {"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb", "uuid":
>> > > > "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
>> {"id":
>> > > "6",
>> > > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
>> > > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
>> {"id":
>> > > "1",
>> > > > "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
>> > > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
>> > > > 2022-08-29 17:01:56,232 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for speed=500Mhz,
>> > > > Ram=1024 MB
>> > > > 2022-08-29 17:01:56,232 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
>> > > > srv-4-comp.aaaaaaaa.bbb, uuid:
>> 134c625d-15ab-461a-a38d-36fb0b979b5d} is
>> > > KVM
>> > > > hypervisor type, no max guest limit check needed
>> > > > 2022-08-29 17:01:56,234 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu
>> capability
>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>> speed:
>> > 500
>> > > > 2022-08-29 17:01:56,234 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has
>> > enough
>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>> > 1073741824 ,
>> > > > cpuOverprovisioningFactor: 1.0
>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
>> > > 127200
>> > > > and CPU after applying overprovisioning: 127200
>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 ,
>> Requested
>> > > > CPU: 500
>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
>> > > > 251009040384 , Requested RAM: (1.00 GB) 1073741824
>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>> RAM
>> > > > available
>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
>> > > host:
>> > > > 5, used: 11100, reserved: 0, actual total: 127200, total with
>> > > > overprovisioning: 127200; requested
>> cpu:500,alloc_from_last_host?:false
>> > > > ,considerReservedCapacity?: true
>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
>> > > host:
>> > > > 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total:
>> (250.27
>> > > GB)
>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>> > alloc_from_last_host?:
>> > > > false , considerReservedCapacity?: true
>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>> > adding
>> > > to
>> > > > list: 5
>> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
>> > > > srv-5-comp.aaaaaaaa.bbb, uuid:
>> 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is
>> > > KVM
>> > > > hypervisor type, no max guest limit check needed
>> > > > 2022-08-29 17:01:56,238 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu
>> capability
>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>> speed:
>> > 500
>> > > > 2022-08-29 17:01:56,238 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has
>> > enough
>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>> > 1073741824 ,
>> > > > cpuOverprovisioningFactor: 1.0
>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
>> > > 127200
>> > > > and CPU after applying overprovisioning: 127200
>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 ,
>> Requested
>> > > CPU:
>> > > > 500
>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
>> > > > 189373743104 , Requested RAM: (1.00 GB) 1073741824
>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>> RAM
>> > > > available
>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
>> > > host:
>> > > > 7, used: 90100, reserved: 0, actual total: 127200, total with
>> > > > overprovisioning: 127200; requested
>> cpu:500,alloc_from_last_host?:false
>> > > > ,considerReservedCapacity?: true
>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
>> > > host:
>> > > > 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total:
>> (250.27
>> > > GB)
>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>> > alloc_from_last_host?:
>> > > > false , considerReservedCapacity?: true
>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>> > adding
>> > > to
>> > > > list: 7
>> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
>> > > > srv-3-comp.aaaaaaaa.bbb, uuid:
>> 6858713b-7e17-46d8-854d-24a13f9c7001} is
>> > > KVM
>> > > > hypervisor type, no max guest limit check needed
>> > > > 2022-08-29 17:01:56,242 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu
>> capability
>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>> speed:
>> > 500
>> > > > 2022-08-29 17:01:56,242 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has
>> > enough
>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>> > 1073741824 ,
>> > > > cpuOverprovisioningFactor: 1.0
>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
>> > > 127200
>> > > > and CPU after applying overprovisioning: 127200
>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 ,
>> Requested
>> > > CPU:
>> > > > 500
>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
>> > > > 204032835584 , Requested RAM: (1.00 GB) 1073741824
>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>> RAM
>> > > > available
>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
>> > > host:
>> > > > 6, used: 42900, reserved: 0, actual total: 127200, total with
>> > > > overprovisioning: 127200; requested
>> cpu:500,alloc_from_last_host?:false
>> > > > ,considerReservedCapacity?: true
>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
>> > > host:
>> > > > 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total:
>> (250.27
>> > > GB)
>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>> > alloc_from_last_host?:
>> > > > false , considerReservedCapacity?: true
>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>> > adding
>> > > to
>> > > > list: 6
>> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
>> > > > srv-2-comp.aaaaaaaa.bbb, uuid:
>> 883f7f62-55f2-42f4-8242-16b6847c4204} is
>> > > KVM
>> > > > hypervisor type, no max guest limit check needed
>> > > > 2022-08-29 17:01:56,246 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu
>> capability
>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>> speed:
>> > 500
>> > > > 2022-08-29 17:01:56,246 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has
>> > enough
>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>> > 1073741824 ,
>> > > > cpuOverprovisioningFactor: 1.0
>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
>> > > 127200
>> > > > and CPU after applying overprovisioning: 127200
>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 ,
>> Requested
>> > > CPU:
>> > > > 500
>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
>> > > > 229802639360 , Requested RAM: (1.00 GB) 1073741824
>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>> RAM
>> > > > available
>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
>> > > host:
>> > > > 1, used: 48200, reserved: 0, actual total: 127200, total with
>> > > > overprovisioning: 127200; requested
>> cpu:500,alloc_from_last_host?:false
>> > > > ,considerReservedCapacity?: true
>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
>> > > host:
>> > > > 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total:
>> (250.27
>> > > GB)
>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>> > alloc_from_last_host?:
>> > > > false , considerReservedCapacity?: true
>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>> > adding
>> > > to
>> > > > list: 1
>> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator returning
>> 4
>> > > > suitable hosts
>> > > > 2022-08-29 17:01:56,258 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>> > > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
>> > > > transited from [Starting] to [Stopped] with event [OperationFailed].
>> > VM's
>> > > > original host: null, new host: null, host before state transition:
>> null
>> > > > 2022-08-29 17:01:56,258 WARN [c.c.v.VirtualMachineManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>> > > > (logid:ba10fec5) Unable to orchestrate start VM instance {id: "85",
>> > name:
>> > > > "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
>> > > > type="ConsoleProxy"} due to [Unable to create deployment, no usable
>> > > volumes
>> > > > found for the VM: 85].
>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to create
>> > > > deployment, no usable volumes found for the VM: 85
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
>> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>> > Source)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>> > > > at
>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> > > > at
>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>> > > > 2022-08-29 17:01:56,259 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>> > > > (logid:ba10fec5) Done executing VM work job:
>> > > >
>> > >
>> >
>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
>> > > > 2022-08-29 17:01:56,260 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>> > > > (logid:ba10fec5) Complete async job-4288, jobStatus: FAILED,
>> > resultCode:
>> > > 0,
>> > > > result:
>> > > >
>> > >
>> >
>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODUiLCBuYW1lOiAidi04NS1WTSIsIHV1aWQ6ICIxMTBkODYyMC1jMDI5LTRjYTktOTA4NC0yMTAwOTk3MjVhODAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODVdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
>> > > > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>> > > > (logid:ba10fec5) Publish async job-4288 complete on message bus
>> > > > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>> > > > (logid:ba10fec5) Wake up jobs related to job-4288
>> > > > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>> > > > (logid:ba10fec5) Update db status for job-4288
>> > > > 2022-08-29 17:01:56,262 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
>> > > > (logid:ba10fec5) Wake up jobs joined with job-4288 and disjoin all
>> > > subjobs
>> > > > created from job- 4288
>> > > > 2022-08-29 17:01:56,270 DEBUG [c.c.v.VmWorkJobDispatcher]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
>> (logid:ba10fec5)
>> > > > Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 85,
>> job
>> > > > origin: 3445
>> > > > 2022-08-29 17:01:56,270 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
>> (logid:ba10fec5)
>> > > > Done executing com.cloud.vm.VmWorkStart for job-4288
>> > > > 2022-08-29 17:01:56,272 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
>> (logid:ba10fec5)
>> > > > Remove job-4288 from job monitoring
>> > > > 2022-08-29 17:01:56,277 WARN [c.c.c.ConsoleProxyManagerImpl]
>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Unable to allocate
>> > console
>> > > > proxy standby capacity for zone [1] due to [Unable to orchestrate
>> start
>> > > VM
>> > > > instance {id: "85", name: "v-85-VM", uuid:
>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>> > > [Unable
>> > > > to create deployment, no usable volumes found for the VM: 85].].
>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to
>> orchestrate
>> > > > start VM instance {id: "85", name: "v-85-VM", uuid:
>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>> > > [Unable
>> > > > to create deployment, no usable volumes found for the VM: 85].
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
>> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>> > Source)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>> > > > at
>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> > > > at
>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>> > > > 2022-08-29 17:01:56,277 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) received console
>> proxy
>> > > alert
>> > > > 2022-08-29 17:01:56,278 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Console proxy
>> creation
>> > > > failure, zone: xxxxxxxxxx
>> > > > 2022-08-29 17:01:56,279 WARN [c.c.a.AlertManagerImpl]
>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) alertType=[10]
>> > > > dataCenterId=[1] podId=[null] clusterId=[null] message=[Console
>> proxy
>> > > > creation failure. zone: xxxxxxxxxx, error details: Unable to
>> > orchestrate
>> > > > start VM instance {id: "85", name: "v-85-VM", uuid:
>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>> > > [Unable
>> > > > to create deployment, no usable volumes found for the VM: 85].].
>> > > > 2022-08-29 17:01:56,284 WARN [c.c.a.AlertManagerImpl]
>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) No recipients set in
>> > > global
>> > > > setting 'alert.email.addresses', skipping sending alert with subject
>> > > > [Console proxy creation failure. zone: xxxxxxxxxx, error details:
>> > Unable
>> > > to
>> > > > orchestrate start VM instance {id: "85", name: "v-85-VM", uuid:
>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>> > > [Unable
>> > > > to create deployment, no usable volumes found for the VM: 85].] and
>> > > content
>> > > > [Console proxy creation failure (zone xxxxxxxxxx)].
>> > > > 2022-08-29 17:01:56,284 WARN [c.c.v.SystemVmLoadScanner]
>> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Unexpected exception
>> > > Unable
>> > > > to orchestrate start VM instance {id: "85", name: "v-85-VM", uuid:
>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>> > > [Unable
>> > > > to create deployment, no usable volumes found for the VM: 85].
>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to
>> orchestrate
>> > > > start VM instance {id: "85", name: "v-85-VM", uuid:
>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>> > > [Unable
>> > > > to create deployment, no usable volumes found for the VM: 85].
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
>> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>> > Source)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>> > > > at
>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> > > > at
>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>> > > > 2022-08-29 17:01:57,721 DEBUG [o.a.c.h.HAManagerImpl]
>> > > > (BackgroundTaskPollManager-4:ctx-9528226f) (logid:4512ff9b) HA
>> health
>> > > check
>> > > > task is running...
>> > > > 2022-08-29 17:01:59,504 DEBUG [c.c.s.StatsCollector]
>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) StorageCollector is
>> > > > running...
>> > > > 2022-08-29 17:01:59,518 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
>> > > getCommandHostDelegation:
>> > > > class com.cloud.agent.api.GetStorageStatsCommand
>> > > > 2022-08-29 17:01:59,518 DEBUG [c.c.h.XenServerGuru]
>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning
>> the
>> > > > default host to execute commands because the command is not of Copy
>> > type.
>> > > > 2022-08-29 17:01:59,571 DEBUG [c.c.a.t.Request]
>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
>> > > > 11-7523826127475836583: Received: { Ans: , MgmtId: 90520740759984,
>> > via:
>> > > > 11(s-32-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
>> > > > 2022-08-29 17:01:59,574 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
>> > > getCommandHostDelegation:
>> > > > class com.cloud.agent.api.GetStorageStatsCommand
>> > > > 2022-08-29 17:01:59,574 DEBUG [c.c.h.XenServerGuru]
>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning
>> the
>> > > > default host to execute commands because the command is not of Copy
>> > type.
>> > > > 2022-08-29 17:01:59,615 DEBUG [c.c.a.t.Request]
>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
>> > > 5-8959067033723473397:
>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>> > > > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>> GetStorageStatsAnswer
>> > > } }
>> > > > 2022-08-29 17:01:59,624 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
>> > > getCommandHostDelegation:
>> > > > class com.cloud.agent.api.GetStorageStatsCommand
>> > > > 2022-08-29 17:01:59,624 DEBUG [c.c.h.XenServerGuru]
>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning
>> the
>> > > > default host to execute commands because the command is not of Copy
>> > type.
>> > > > 2022-08-29 17:01:59,653 DEBUG [c.c.a.t.Request]
>> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
>> > > 6-4833769775052102466:
>> > > > Received: { Ans: , MgmtId: 90520740759984, via:
>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
>> GetStorageStatsAnswer
>> > > } }
>> > > > 2022-08-29 17:01:59,979 DEBUG [c.c.a.m.AgentManagerImpl]
>> > > > (AgentManager-Handler-12:null) (logid:) Ping from Routing host
>> > > > 5(srv-4-comp.aaaaaaaa.bbb)
>> > > >
>> > > > 2022-08-29 17:02:00,269 DEBUG [c.c.s.StatsCollector]
>> > > > (StatsCollector-5:ctx-d2b6ffd3) (logid:95201678) DbCollector is
>> > > running...
>> > > > 2022-08-29 17:02:00,443 DEBUG
>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
>> > > > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 2
>> routers
>> > to
>> > > > update status.
>> > > > 2022-08-29 17:02:00,449 DEBUG
>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
>> > > > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 1 VPC's
>> to
>> > > > update Redundant State.
>> > > > 2022-08-29 17:02:00,450 DEBUG
>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
>> > > > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 0
>> networks
>> > to
>> > > > update RvR status.
>> > > > 2022-08-29 17:02:00,457 DEBUG [c.c.a.t.Request]
>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
>> > > > 6-4833769775052102467: Sending { Cmd , MgmtId: 90520740759984, via:
>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
>> > > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
>> > > router.name
>> > >
>> >
>> ":"r-79-VM","router.ip":"169.254.144.198"},"wait":"30","bypassHostMaintenance":"false"}}]
>> > > > }
>> > > > 2022-08-29 17:02:00,599 DEBUG [c.c.a.t.Request]
>> > > > (AgentManager-Handler-7:null) (logid:) Seq 6-4833769775052102467:
>> > > > Processing: { Ans: , MgmtId: 90520740759984, via: 6, Ver: v1,
>> Flags:
>> > 10,
>> > > >
>> > >
>> >
>> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"PRIMARY","result":"true","details":"Status:
>> > > > PRIMARY
>> > > > ","wait":"0","bypassHostMaintenance":"false"}}] }
>> > > > 2022-08-29 17:02:00,599 DEBUG [c.c.a.t.Request]
>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
>> > > > 6-4833769775052102467: Received: { Ans: , MgmtId: 90520740759984,
>> via:
>> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer
>> } }
>> > > > 2022-08-29 17:02:00,599 DEBUG [c.c.a.m.AgentManagerImpl]
>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f)
>> Details
>> > > > from executing class com.cloud.agent.api.CheckRouterCommand: Status:
>> > > PRIMARY
>> > > >
>> > > > 2022-08-29 17:02:00,612 DEBUG [c.c.a.t.Request]
>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
>> > > > 1-5410793477308947404: Sending { Cmd , MgmtId: 90520740759984, via:
>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
>> > > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
>> > > router.name
>> > >
>> >
>> ":"r-80-VM","router.ip":"169.254.217.200"},"wait":"30","bypassHostMaintenance":"false"}}]
>> > > > }
>> > > > 2022-08-29 17:02:00,748 DEBUG [c.c.a.m.AgentManagerImpl]
>> > > > (AgentManager-Handler-4:null) (logid:) Ping from 11(s-32-VM)
>> > > > 2022-08-29 17:02:00,750 DEBUG [c.c.a.t.Request]
>> > > > (AgentManager-Handler-1:null) (logid:) Seq 1-5410793477308947404:
>> > > > Processing: { Ans: , MgmtId: 90520740759984, via: 1, Ver: v1,
>> Flags:
>> > 10,
>> > > >
>> > >
>> >
>> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"BACKUP","result":"true","details":"Status:
>> > > > BACKUP
>> > > > ","wait":"0","bypassHostMaintenance":"false"}}] }
>> > > > 2022-08-29 17:02:00,750 DEBUG [c.c.a.t.Request]
>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
>> > > > 1-5410793477308947404: Received: { Ans: , MgmtId: 90520740759984,
>> via:
>> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer
>> } }
>> > > > 2022-08-29 17:02:00,751 DEBUG [c.c.a.m.AgentManagerImpl]
>> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f)
>> Details
>> > > > from executing class com.cloud.agent.api.CheckRouterCommand: Status:
>> > > BACKUP
>> > > >
>> > > > 2022-08-29 17:02:01,362 DEBUG [c.c.s.StatsCollector]
>> > > > (Cluster-Worker-839:ctx-e097a8ab) (logid:3deacf65) StatusUpdate from
>> > > > 90520740759984, json:
>> > > >
>> > >
>> >
>> {"managementServerHostId":1,"managementServerHostUuid":"5615ca5f-0716-4da1-93ec-31f79fd19bbe","collectionTime":"Aug
>> > > > 29, 2022, 5:02:01
>> > > >
>> > >
>> >
>> PM","sessions":1,"cpuUtilization":0.0,"totalJvmMemoryBytes":324534272,"freeJvmMemoryBytes":68595424,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":103760388,"jvmStartTime":1661681560910,"availableProcessors":6,"loadAverage":0.02,"totalInit":139788288,"totalUsed":474791104,"totalCommitted":557662208,"pid":1522332,"jvmName":"1522332@vm-1-cs-mgmt.aaaaaaaa.bbb
>> > >
>> >
>> ","jvmVendor":"Ubuntu","jvmVersion":"11.0.16+8-post-Ubuntu-0ubuntu120.04","osDistribution":"Ubuntu
>> > > > 20.04.5
>> > > >
>> > >
>> >
>> LTS","agentCount":5,"heapMemoryUsed":256775312,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":13,"threadsRunnableCount":15,"threadsTerminatedCount":0,"threadsTotalCount":420,"threadsWaitingCount":331,"systemMemoryTotal":8343236608,"systemMemoryFree":2623193088,"systemMemoryUsed":814808,"systemMemoryVirtualSize":7494184960,"logInfo":"/var/log/cloudstack/management/management-server.log
>> > > > using: 74M\non disk /dev/vda2 mounted on / (14%
>> > > >
>> > >
>> >
>> full)","systemTotalCpuCycles":15870.180000000002,"systemLoadAverages":[0.02,0.03,0.0],"systemCyclesUsage":[4116569,2615101,2716460424],"dbLocal":false,"usageLocal":false,"systemBootTime":"Jul
>> > > > 8, 2022, 2:35:55 AM","kernelVersion":"5.4.0-121-generic"}
>> > > > 2022-08-29 17:02:01,723 DEBUG [o.a.c.h.HAManagerImpl]
>> > > > (BackgroundTaskPollManager-4:ctx-7abb1b5d) (logid:2320950d) HA
>> health
>> > > check
>> > > > task is running...
>> > > > 2022-08-29 17:02:05,417 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> > > > (Vm-Operations-Cleanup-1:ctx-e741e208) (logid:3f5129e0) VM Operation
>> > > Thread
>> > > > Running
>> > > > 2022-08-29 17:02:05,725 DEBUG [o.a.c.h.HAManagerImpl]
>> > > > (BackgroundTaskPollManager-4:ctx-8e7f6766) (logid:710cb3b2) HA
>> health
>> > > check
>> > > > task is running...
>> > > > 2022-08-29 17:02:08,626 DEBUG [c.c.a.m.AgentManagerImpl]
>> > > > (AgentManager-Handler-11:null) (logid:) Ping from Routing host
>> > > > 6(srv-3-comp.aaaaaaaa.bbb)
>> > > >
>> > > > 2022-08-29 17:02:09,727 DEBUG [o.a.c.h.HAManagerImpl]
>> > > > (BackgroundTaskPollManager-4:ctx-26a5ff07) (logid:09b3c89a) HA
>> health
>> > > check
>> > > > task is running...
>> > > > 2022-08-29 17:02:13,729 DEBUG [o.a.c.h.HAManagerImpl]
>> > > > (BackgroundTaskPollManager-4:ctx-b4d4e374) (logid:9048c7a7) HA
>> health
>> > > check
>> > > > task is running...
>> > > > 2022-08-29 17:02:17,731 DEBUG [o.a.c.h.HAManagerImpl]
>> > > > (BackgroundTaskPollManager-4:ctx-3db3d208) (logid:7bb73f7b) HA
>> health
>> > > check
>> > > > task is running...
>> > > > 2022-08-29 17:02:21,734 DEBUG [o.a.c.h.HAManagerImpl]
>> > > > (BackgroundTaskPollManager-1:ctx-fca1d248) (logid:1ed5381a) HA
>> health
>> > > check
>> > > > task is running...
>> > > > 2022-08-29 17:02:25,737 DEBUG [o.a.c.h.HAManagerImpl]
>> > > > (BackgroundTaskPollManager-1:ctx-984e2382) (logid:e6c3d724) HA
>> health
>> > > check
>> > > > task is running...
>> > > > 2022-08-29 17:02:25,759 DEBUG [c.c.s.StatsCollector]
>> > > > (secstorage-1:ctx-64c610ed) (logid:6603cd38) Verifying image storage
>> > [1].
>> > > > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
>> > > > 2022-08-29 17:02:25,763 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
>> > > > (secstorage-1:ctx-64c610ed) (logid:6603cd38) Zone [1] is ready to
>> > launch
>> > > > secondary storage VM.
>> > > > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Zone 1 is ready to
>> > launch
>> > > > console proxy
>> > > > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Expand console proxy
>> > > standby
>> > > > capacity for zone xxxxxxxxxx
>> > > > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Allocating console
>> proxy
>> > > > standby capacity for zone [1].
>> > > > 2022-08-29 17:02:26,030 INFO [c.c.c.ConsoleProxyManagerImpl]
>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Found a stopped
>> console
>> > > > proxy, starting it. Vm id : 85
>> > > > 2022-08-29 17:02:26,031 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) start parameter
>> value of
>> > > > enterHardwareSetup == <very null> during processing of queued job
>> > > > 2022-08-29 17:02:26,037 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Sync job-4289
>> execution
>> > on
>> > > > object VmWorkJobQueue.85
>> > > > 2022-08-29 17:02:26,167 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (AsyncJobMgr-Heartbeat-1:ctx-bba6b89d) (logid:f9aed0eb) Execute
>> > > sync-queue
>> > > > item: SyncQueueItemVO {id:2106, queueId: 2105, contentType:
>> AsyncJob,
>> > > > contentId: 4289, lastProcessMsid: 90520740759984,
>> lastprocessNumber: 2,
>> > > > lastProcessTime: Mon Aug 29 17:02:26 CEST 2022, created: Mon Aug 29
>> > > > 17:02:26 CEST 2022}
>> > > > 2022-08-29 17:02:26,167 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (AsyncJobMgr-Heartbeat-1:ctx-bba6b89d) (logid:f9aed0eb) Schedule
>> queued
>> > > > job-4289
>> > > > 2022-08-29 17:02:26,172 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
>> (logid:f919f9b3)
>> > > Add
>> > > > job-4289 into job monitoring
>> > > > 2022-08-29 17:02:26,181 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
>> (logid:ba10fec5)
>> > > > Executing AsyncJobVO: {id:4289, userId: 1, accountId: 1,
>> instanceType:
>> > > > null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
>> > > >
>> > >
>> >
>> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAVXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
>> > > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>> > > > result: null, initMsid: 90520740759984, completeMsid: null,
>> > lastUpdated:
>> > > > null, lastPolled: null, created: Mon Aug 29 17:02:26 CEST 2022,
>> > removed:
>> > > > null}
>> > > > 2022-08-29 17:02:26,181 DEBUG [c.c.v.VmWorkJobDispatcher]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
>> (logid:ba10fec5)
>> > > Run
>> > > > VM work job: com.cloud.vm.VmWorkStart for VM 85, job origin: 3445
>> > > > 2022-08-29 17:02:26,183 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>> > > > (logid:ba10fec5) Execute VM work job:
>> > > >
>> > >
>> >
>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
>> > > > 2022-08-29 17:02:26,184 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>> > > > (logid:ba10fec5) orchestrating VM start for 'v-85-VM'
>> > > > com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
>> > > > 2022-08-29 17:02:26,192 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>> > > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
>> > > > transited from [Stopped] to [Starting] with event [StartRequested].
>> > VM's
>> > > > original host: null, new host: null, host before state transition:
>> null
>> > > > 2022-08-29 17:02:26,192 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>> > > > (logid:ba10fec5) Successfully transitioned to start state for VM
>> > instance
>> > > > {id: "85", name: "v-85-VM", uuid:
>> > "110d8620-c029-4ca9-9084-210099725a80",
>> > > > type="ConsoleProxy"} reservation id =
>> > > 24bf955f-96d8-4f4f-ae33-2cae2f3e0425
>> > > > 2022-08-29 17:02:26,198 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>> > > > (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts:
>> null
>> > > > 2022-08-29 17:02:26,200 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>> > > > (logid:ba10fec5) VM start attempt #1
>> > > > 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>> > > > (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
>> > > > 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>> > > > (logid:ba10fec5) Trying to allocate a host and storage pools from
>> dc:1,
>> > > > pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
>> > > > 1073741824
>> > > > 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>> > > > (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
>> > > > 2022-08-29 17:02:26,211 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>> > > > (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
>> > > > 2022-08-29 17:02:26,212 DEBUG [c.c.d.FirstFitPlanner]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>> > > > (logid:ba10fec5) Searching all possible resources under this Zone: 1
>> > > > 2022-08-29 17:02:26,213 DEBUG [c.c.d.FirstFitPlanner]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>> > > > (logid:ba10fec5) Listing clusters in order of aggregate capacity,
>> that
>> > > have
>> > > > (at least one host with) enough CPU and RAM capacity under this
>> Zone: 1
>> > > > 2022-08-29 17:02:26,216 DEBUG [c.c.d.FirstFitPlanner]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>> > > > (logid:ba10fec5) Removing from the clusterId list these clusters
>> from
>> > > avoid
>> > > > set: []
>> > > > 2022-08-29 17:02:26,221 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>> > > > (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
>> > > > 2022-08-29 17:02:26,222 INFO [c.c.a.m.a.i.FirstFitAllocator]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested
>> with
>> > > > Custom[UEFI] Boot Type false
>> > > > 2022-08-29 17:02:26,223 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in dc:
>> 1
>> > > > pod:1 cluster:1
>> > > > 2022-08-29 17:02:26,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4
>> > hosts
>> > > to
>> > > > check for allocation: [Host {"id": "7", "name":
>> > > "srv-5-comp.aaaaaaaa.bbb",
>> > > > "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"},
>> Host
>> > > > {"id": "1", "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
>> > > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}, Host
>> {"id":
>> > > "6",
>> > > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
>> > > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
>> {"id":
>> > > "5",
>> > > > "name": "srv-4-comp.aaaaaaaa.bbb", "uuid":
>> > > > "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}]
>> > > > 2022-08-29 17:02:26,231 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for
>> allocation
>> > > > after prioritization: [Host {"id": "7", "name":
>> > > "srv-5-comp.aaaaaaaa.bbb",
>> > > > "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"},
>> Host
>> > > > {"id": "1", "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
>> > > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}, Host
>> {"id":
>> > > "6",
>> > > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
>> > > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
>> {"id":
>> > > "5",
>> > > > "name": "srv-4-comp.aaaaaaaa.bbb", "uuid":
>> > > > "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}]
>> > > > 2022-08-29 17:02:26,231 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for speed=500Mhz,
>> > > > Ram=1024 MB
>> > > > 2022-08-29 17:02:26,231 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
>> > > > srv-5-comp.aaaaaaaa.bbb, uuid:
>> 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is
>> > > KVM
>> > > > hypervisor type, no max guest limit check needed
>> > > > 2022-08-29 17:02:26,234 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu
>> capability
>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>> speed:
>> > 500
>> > > > 2022-08-29 17:02:26,234 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has
>> > enough
>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>> > 1073741824 ,
>> > > > cpuOverprovisioningFactor: 1.0
>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
>> > > 127200
>> > > > and CPU after applying overprovisioning: 127200
>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 ,
>> Requested
>> > > CPU:
>> > > > 500
>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
>> > > > 189373743104 , Requested RAM: (1.00 GB) 1073741824
>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>> RAM
>> > > > available
>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
>> > > host:
>> > > > 7, used: 90100, reserved: 0, actual total: 127200, total with
>> > > > overprovisioning: 127200; requested
>> cpu:500,alloc_from_last_host?:false
>> > > > ,considerReservedCapacity?: true
>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
>> > > host:
>> > > > 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total:
>> (250.27
>> > > GB)
>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>> > alloc_from_last_host?:
>> > > > false , considerReservedCapacity?: true
>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>> > adding
>> > > to
>> > > > list: 7
>> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
>> > > > srv-2-comp.aaaaaaaa.bbb, uuid:
>> 883f7f62-55f2-42f4-8242-16b6847c4204} is
>> > > KVM
>> > > > hypervisor type, no max guest limit check needed
>> > > > 2022-08-29 17:02:26,239 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu
>> capability
>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>> speed:
>> > 500
>> > > > 2022-08-29 17:02:26,239 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has
>> > enough
>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>> > 1073741824 ,
>> > > > cpuOverprovisioningFactor: 1.0
>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
>> > > 127200
>> > > > and CPU after applying overprovisioning: 127200
>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 ,
>> Requested
>> > > CPU:
>> > > > 500
>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
>> > > > 229802639360 , Requested RAM: (1.00 GB) 1073741824
>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>> RAM
>> > > > available
>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
>> > > host:
>> > > > 1, used: 48200, reserved: 0, actual total: 127200, total with
>> > > > overprovisioning: 127200; requested
>> cpu:500,alloc_from_last_host?:false
>> > > > ,considerReservedCapacity?: true
>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
>> > > host:
>> > > > 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total:
>> (250.27
>> > > GB)
>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>> > alloc_from_last_host?:
>> > > > false , considerReservedCapacity?: true
>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>> > adding
>> > > to
>> > > > list: 1
>> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
>> > > > srv-3-comp.aaaaaaaa.bbb, uuid:
>> 6858713b-7e17-46d8-854d-24a13f9c7001} is
>> > > KVM
>> > > > hypervisor type, no max guest limit check needed
>> > > > 2022-08-29 17:02:26,243 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu
>> capability
>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>> speed:
>> > 500
>> > > > 2022-08-29 17:02:26,243 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has
>> > enough
>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>> > 1073741824 ,
>> > > > cpuOverprovisioningFactor: 1.0
>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
>> > > 127200
>> > > > and CPU after applying overprovisioning: 127200
>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 ,
>> Requested
>> > > CPU:
>> > > > 500
>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
>> > > > 204032835584 , Requested RAM: (1.00 GB) 1073741824
>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>> RAM
>> > > > available
>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
>> > > host:
>> > > > 6, used: 42900, reserved: 0, actual total: 127200, total with
>> > > > overprovisioning: 127200; requested
>> cpu:500,alloc_from_last_host?:false
>> > > > ,considerReservedCapacity?: true
>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
>> > > host:
>> > > > 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total:
>> (250.27
>> > > GB)
>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>> > alloc_from_last_host?:
>> > > > false , considerReservedCapacity?: true
>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>> > adding
>> > > to
>> > > > list: 6
>> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
>> > > > srv-4-comp.aaaaaaaa.bbb, uuid:
>> 134c625d-15ab-461a-a38d-36fb0b979b5d} is
>> > > KVM
>> > > > hypervisor type, no max guest limit check needed
>> > > > 2022-08-29 17:02:26,247 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu
>> capability
>> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested
>> speed:
>> > 500
>> > > > 2022-08-29 17:02:26,247 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has
>> > enough
>> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>> > 1073741824 ,
>> > > > cpuOverprovisioningFactor: 1.0
>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
>> > > 127200
>> > > > and CPU after applying overprovisioning: 127200
>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 ,
>> Requested
>> > > > CPU: 500
>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
>> > > > 251009040384 , Requested RAM: (1.00 GB) 1073741824
>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
>> RAM
>> > > > available
>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
>> > > host:
>> > > > 5, used: 11100, reserved: 0, actual total: 127200, total with
>> > > > overprovisioning: 127200; requested
>> cpu:500,alloc_from_last_host?:false
>> > > > ,considerReservedCapacity?: true
>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
>> > > host:
>> > > > 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total:
>> (250.27
>> > > GB)
>> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
>> > alloc_from_last_host?:
>> > > > false , considerReservedCapacity?: true
>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
>> > adding
>> > > to
>> > > > list: 5
>> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
>> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator returning
>> 4
>> > > > suitable hosts
>> > > > 2022-08-29 17:02:26,260 DEBUG [c.c.c.CapacityManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>> > > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
>> > > > transited from [Starting] to [Stopped] with event [OperationFailed].
>> > VM's
>> > > > original host: null, new host: null, host before state transition:
>> null
>> > > > 2022-08-29 17:02:26,260 WARN [c.c.v.VirtualMachineManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>> > > > (logid:ba10fec5) Unable to orchestrate start VM instance {id: "85",
>> > name:
>> > > > "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
>> > > > type="ConsoleProxy"} due to [Unable to create deployment, no usable
>> > > volumes
>> > > > found for the VM: 85].
>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to create
>> > > > deployment, no usable volumes found for the VM: 85
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
>> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>> > Source)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>> > > > at
>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> > > > at
>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>> > > > 2022-08-29 17:02:26,261 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>> > > > (logid:ba10fec5) Done executing VM work job:
>> > > >
>> > >
>> >
>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
>> > > > 2022-08-29 17:02:26,262 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>> > > > (logid:ba10fec5) Complete async job-4289, jobStatus: FAILED,
>> > resultCode:
>> > > 0,
>> > > > result:
>> > > >
>> > >
>> >
>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODUiLCBuYW1lOiAidi04NS1WTSIsIHV1aWQ6ICIxMTBkODYyMC1jMDI5LTRjYTktOTA4NC0yMTAwOTk3MjVhODAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODVdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
>> > > > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>> > > > (logid:ba10fec5) Publish async job-4289 complete on message bus
>> > > > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>> > > > (logid:ba10fec5) Wake up jobs related to job-4289
>> > > > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>> > > > (logid:ba10fec5) Update db status for job-4289
>> > > > 2022-08-29 17:02:26,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
>> > > > (logid:ba10fec5) Wake up jobs joined with job-4289 and disjoin all
>> > > subjobs
>> > > > created from job- 4289
>> > > > 2022-08-29 17:02:26,269 DEBUG [c.c.v.VmWorkJobDispatcher]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
>> (logid:ba10fec5)
>> > > > Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 85,
>> job
>> > > > origin: 3445
>> > > > 2022-08-29 17:02:26,270 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
>> (logid:ba10fec5)
>> > > > Done executing com.cloud.vm.VmWorkStart for job-4289
>> > > > 2022-08-29 17:02:26,271 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
>> (logid:ba10fec5)
>> > > > Remove job-4289 from job monitoring
>> > > > 2022-08-29 17:02:26,277 WARN [c.c.c.ConsoleProxyManagerImpl]
>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Unable to allocate
>> > console
>> > > > proxy standby capacity for zone [1] due to [Unable to orchestrate
>> start
>> > > VM
>> > > > instance {id: "85", name: "v-85-VM", uuid:
>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>> > > [Unable
>> > > > to create deployment, no usable volumes found for the VM: 85].].
>> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to
>> orchestrate
>> > > > start VM instance {id: "85", name: "v-85-VM", uuid:
>> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
>> > > [Unable
>> > > > to create deployment, no usable volumes found for the VM: 85].
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
>> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>> > Source)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>> > > > at
>> > > >
>> > >
>> >
>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>> > > > at
>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>> > > > at
>> > > >
>> > >
>> >
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> > > > at
>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> > > > at
>> > > >
>> > >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> > > > at java.base/java.lang.Thread.run(Thread.java:829)
>> > > > 2022-08-29 17:02:26,277 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) received console
>> proxy
>> > > alert
>> > > >
>> > > >
>> > > >
>> > > > Am Mo., 29. Aug. 2022 um 16:30 Uhr schrieb Slavka Peleva <
>> > > > slavkap@storpool.com>:
>> > > >
>> > > >> Thanks, Chris, for the information!
>> > > >>
>> > > >> Are you using custom offerings for the system VMs? Can you share
>> more
>> > > >> info if yes?
>> > > >>
>> > > >> About the log, it's not the complete log from where the deployment
>> > > >> started. The deployment begins a bit earlier than what you shared.
>> Can
>> > > you
>> > > >> search the management log for `Sync job-3539 execution on object
>> > > >> VmWorkJobQueue` to get the `logid` for the volume creation
>> (coloured
>> > in
>> > > >> blue)? And if possible, share the logs with the found log ID
>> > > >>
>> > > >> Here is an example from my logs:
>> > > >>
>> > > >> 2022-08-29 17:07:49,114 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > >> (consoleproxy-1:ctx-a952a3dc) (logid:89157bb9) Sync job-60484
>> > execution
>> > > >> on object VmWorkJobQueue.4904
>> > > >>
>> > > >> 2022-08-29 17:07:50,816 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>> > > >> (Work-Job-Executor-5:ctx-56172581 job-60479/job-60484 ctx-a900986d)
>> > > >> (logid:95d750ae) Execute VM work job:
>> > > >>
>> > >
>> >
>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":4904,"handlerName":"VirtualMachineManagerImpl"}
>> > > >> 2022-08-29 17:07:50,817 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> > > >> (Work-Job-Executor-5:ctx-56172581 job-60479/job-60484 ctx-a900986d)
>> > > >> (logid:95d750ae) orchestrating VM start for 'v-4904-VM'
>> > > >> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
>> > > >>
>> > > >>
>> > > >> On Mon, Aug 29, 2022 at 2:29 PM Vash_X@gmx.de <Va...@gmx.de>
>> wrote:
>> > > >>
>> > > >>> Hi Slavka,
>> > > >>>
>> > > >>> thats what i did after your suggestion / informatin and where the
>> > error
>> > > >>> occured.
>> > > >>> Deleted / Destroyed the consoleproxy and waited till recreation.
>> > Which
>> > > >>> didn't worked throwing the error i pasted above.
>> > > >>> Sorry for beeing so unspecific.
>> > > >>>
>> > > >>> Regards,
>> > > >>> Chris
>> > > >>>
>> > > >>> Am Mo., 29. Aug. 2022 um 12:46 Uhr schrieb Slavka Peleva <
>> > > >>> slavkap@storpool.com>:
>> > > >>>
>> > > >>>> Hi Chris,
>> > > >>>>
>> > > >>>> I mean to recreate the system VMs by destroying them.
>> > > >>>>
>> > > >>>> On Mon, Aug 29, 2022 at 12:38 PM Vash_X@gmx.de <Va...@gmx.de>
>> > wrote:
>> > > >>>>
>> > > >>>>> Hi Slavka,
>> > > >>>>>
>> > > >>>>> didn't tried to restart the SystemVMs. Reading through the docs
>> i
>> > had
>> > > >>>>> the impression that "live-patching" would be enough for this....
>> > > >>>>> Anyway - I tried my luck with the console proxy.
>> > > >>>>>
>> > > >>>>> Which failed utterly. Not starting as it isn't finding any
>> useable
>> > > >>>>> storage pool. - logs at the end of the mail.
>> > > >>>>>
>> > > >>>>> I tried deploying a VM using the same storage pool which worked
>> > > >>>>> wonderfully.
>> > > >>>>> There i noticed that all my custome storage-offerings are
>> unuseable
>> > > at
>> > > >>>>> the moment. They are not accepting any custome storage sizes
>> > > anymore...
>> > > >>>>> Throwing failures "VM Creation failed. Volume size: -1GB is out
>> of
>> > > allowed
>> > > >>>>> tange. Max:3072 Min:1"...
>> > > >>>>> So this is also currently broken....
>> > > >>>>>
>> > > >>>>> 2022-08-29 10:51:56,189 DEBUG [c.c.v.VmWorkJobDispatcher]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
>> > > (logid:ba10fec5) Run
>> > > >>>>> VM work job: com.cloud.vm.VmWorkStart for VM 83, job origin:
>> 3445
>> > > >>>>> 2022-08-29 10:51:56,191 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751)
>> > > >>>>> (logid:ba10fec5) Execute VM work job:
>> > > >>>>>
>> > >
>> >
>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
>> > > >>>>> 2022-08-29 10:51:56,192 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751)
>> > > >>>>> (logid:ba10fec5) orchestrating VM start for 'v-83-VM'
>> > > >>>>> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
>> > > >>>>> 2022-08-29 10:51:56,198 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751)
>> > > >>>>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
>> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
>> state
>> > > >>>>> transited from [Stopped] to [Starting] with event
>> [StartRequested].
>> > > VM's
>> > > >>>>> original host: null, new host: null, host before state
>> transition:
>> > > null
>> > > >>>>> 2022-08-29 10:51:56,198 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751)
>> > > >>>>> (logid:ba10fec5) Successfully transitioned to start state for VM
>> > > instance
>> > > >>>>> {id: "83", name: "v-83-VM", uuid:
>> > > "b1d1f9c6-eafe-4527-9335-e649c646aab0",
>> > > >>>>> type="ConsoleProxy"} reservation id =
>> > > 6641ccc0-bcef-4b1a-a1f7-8dd8000626ad
>> > > >>>>> 2022-08-29 10:51:56,203 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751)
>> > > >>>>> (logid:ba10fec5) Deploy avoids pods: null, clusters: null,
>> hosts:
>> > > null
>> > > >>>>> 2022-08-29 10:51:56,206 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751)
>> > > >>>>> (logid:ba10fec5) VM start attempt #1
>> > > >>>>> 2022-08-29 10:51:56,207 DEBUG
>> [c.c.d.DeploymentPlanningManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751)
>> > > >>>>> (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
>> > > >>>>> 2022-08-29 10:51:56,207 DEBUG
>> [c.c.d.DeploymentPlanningManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751)
>> > > >>>>> (logid:ba10fec5) Trying to allocate a host and storage pools
>> from
>> > > dc:1,
>> > > >>>>> pod:null,cluster:null, requested cpu: 500, requested ram: (1.00
>> GB)
>> > > >>>>> 1073741824
>> > > >>>>> 2022-08-29 10:51:56,207 DEBUG
>> [c.c.d.DeploymentPlanningManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751)
>> > > >>>>> (logid:ba10fec5) Is ROOT volume READY (pool already
>> allocated)?: No
>> > > >>>>> 2022-08-29 10:51:56,214 DEBUG
>> [c.c.d.DeploymentPlanningManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751)
>> > > >>>>> (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts:
>> null
>> > > >>>>> 2022-08-29 10:51:56,215 DEBUG [c.c.d.FirstFitPlanner]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751)
>> > > >>>>> (logid:ba10fec5) Searching all possible resources under this
>> Zone:
>> > 1
>> > > >>>>> 2022-08-29 10:51:56,216 DEBUG [c.c.d.FirstFitPlanner]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751)
>> > > >>>>> (logid:ba10fec5) Listing clusters in order of aggregate
>> capacity,
>> > > that have
>> > > >>>>> (at least one host with) enough CPU and RAM capacity under this
>> > > Zone: 1
>> > > >>>>> 2022-08-29 10:51:56,219 DEBUG [c.c.d.FirstFitPlanner]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751)
>> > > >>>>> (logid:ba10fec5) Removing from the clusterId list these clusters
>> > > from avoid
>> > > >>>>> set: []
>> > > >>>>> 2022-08-29 10:51:56,224 DEBUG
>> [c.c.d.DeploymentPlanningManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751)
>> > > >>>>> (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
>> > > >>>>> 2022-08-29 10:51:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is
>> requested
>> > > with
>> > > >>>>> Custom[UEFI] Boot Type false
>> > > >>>>> 2022-08-29 10:51:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in
>> > dc: 1
>> > > >>>>> pod:1 cluster:1
>> > > >>>>> 2022-08-29 10:51:56,227 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator
>> has 4
>> > > hosts to
>> > > >>>>> check for allocation: [Host {"id": "7", "name":
>> > > "srv-5-comp.blabla.blub",
>> > > >>>>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6",
>> "type"="Routing"},
>> > > Host
>> > > >>>>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
>> > > >>>>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
>> > > {"id": "6",
>> > > >>>>> "name": "srv-3-comp.blabla.blub", "uuid":
>> > > >>>>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
>> > > {"id": "1",
>> > > >>>>> "name": "srv-2-comp.blabla.blub", "uuid":
>> > > >>>>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
>> > > >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for
>> > > allocation
>> > > >>>>> after prioritization: [Host {"id": "7", "name":
>> > > "srv-5-comp.blabla.blub",
>> > > >>>>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6",
>> "type"="Routing"},
>> > > Host
>> > > >>>>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
>> > > >>>>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
>> > > {"id": "6",
>> > > >>>>> "name": "srv-3-comp.blabla.blub", "uuid":
>> > > >>>>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
>> > > {"id": "1",
>> > > >>>>> "name": "srv-2-comp.blabla.blub", "uuid":
>> > > >>>>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
>> > > >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for
>> > speed=500Mhz,
>> > > >>>>> Ram=1024 MB
>> > > >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
>> > > >>>>> srv-5-comp.blabla.blub, uuid:
>> 8c753ce1-0253-49c9-9c7f-1f971a719ce6}
>> > > is KVM
>> > > >>>>> hypervisor type, no max guest limit check needed
>> > > >>>>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu
>> > capability
>> > > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
>> > > speed: 500
>> > > >>>>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7
>> has
>> > > enough
>> > > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>> > > 1073741824 ,
>> > > >>>>> cpuOverprovisioningFactor: 1.0
>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>> > CPU:
>> > > 127200
>> > > >>>>> and CPU after applying overprovisioning: 127200
>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 ,
>> > > Requested CPU:
>> > > >>>>> 500
>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
>> > > >>>>> 189373743104 , Requested RAM: (1.00 GB) 1073741824
>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
>> and
>> > > RAM
>> > > >>>>> available
>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>> > from
>> > > host:
>> > > >>>>> 7, used: 90100, reserved: 0, actual total: 127200, total with
>> > > >>>>> overprovisioning: 127200; requested
>> > > cpu:500,alloc_from_last_host?:false
>> > > >>>>> ,considerReservedCapacity?: true
>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>> > from
>> > > host:
>> > > >>>>> 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total:
>> > > (250.27 GB)
>> > > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
>> > > alloc_from_last_host?:
>> > > >>>>> false , considerReservedCapacity?: true
>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable
>> host,
>> > > adding to
>> > > >>>>> list: 7
>> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
>> > > >>>>> srv-4-comp.blabla.blub, uuid:
>> 134c625d-15ab-461a-a38d-36fb0b979b5d}
>> > > is KVM
>> > > >>>>> hypervisor type, no max guest limit check needed
>> > > >>>>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu
>> > capability
>> > > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
>> > > speed: 500
>> > > >>>>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5
>> has
>> > > enough
>> > > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>> > > 1073741824 ,
>> > > >>>>> cpuOverprovisioningFactor: 1.0
>> > > >>>>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>> > CPU:
>> > > 127200
>> > > >>>>> and CPU after applying overprovisioning: 127200
>> > > >>>>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 ,
>> > > Requested
>> > > >>>>> CPU: 500
>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
>> > > >>>>> 251009040384 , Requested RAM: (1.00 GB) 1073741824
>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
>> and
>> > > RAM
>> > > >>>>> available
>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>> > from
>> > > host:
>> > > >>>>> 5, used: 11100, reserved: 0, actual total: 127200, total with
>> > > >>>>> overprovisioning: 127200; requested
>> > > cpu:500,alloc_from_last_host?:false
>> > > >>>>> ,considerReservedCapacity?: true
>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>> > from
>> > > host:
>> > > >>>>> 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total:
>> > > (250.27 GB)
>> > > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
>> > > alloc_from_last_host?:
>> > > >>>>> false , considerReservedCapacity?: true
>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable
>> host,
>> > > adding to
>> > > >>>>> list: 5
>> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
>> > > >>>>> srv-3-comp.blabla.blub, uuid:
>> 6858713b-7e17-46d8-854d-24a13f9c7001}
>> > > is KVM
>> > > >>>>> hypervisor type, no max guest limit check needed
>> > > >>>>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu
>> > capability
>> > > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
>> > > speed: 500
>> > > >>>>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6
>> has
>> > > enough
>> > > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>> > > 1073741824 ,
>> > > >>>>> cpuOverprovisioningFactor: 1.0
>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>> > CPU:
>> > > 127200
>> > > >>>>> and CPU after applying overprovisioning: 127200
>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 ,
>> > > Requested CPU:
>> > > >>>>> 500
>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
>> > > >>>>> 204032835584 , Requested RAM: (1.00 GB) 1073741824
>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
>> and
>> > > RAM
>> > > >>>>> available
>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>> > from
>> > > host:
>> > > >>>>> 6, used: 42900, reserved: 0, actual total: 127200, total with
>> > > >>>>> overprovisioning: 127200; requested
>> > > cpu:500,alloc_from_last_host?:false
>> > > >>>>> ,considerReservedCapacity?: true
>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>> > from
>> > > host:
>> > > >>>>> 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total:
>> > > (250.27 GB)
>> > > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
>> > > alloc_from_last_host?:
>> > > >>>>> false , considerReservedCapacity?: true
>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable
>> host,
>> > > adding to
>> > > >>>>> list: 6
>> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
>> > > >>>>> srv-2-comp.blabla.blub, uuid:
>> 883f7f62-55f2-42f4-8242-16b6847c4204}
>> > > is KVM
>> > > >>>>> hypervisor type, no max guest limit check needed
>> > > >>>>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu
>> > capability
>> > > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
>> > > speed: 500
>> > > >>>>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1
>> has
>> > > enough
>> > > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
>> > > 1073741824 ,
>> > > >>>>> cpuOverprovisioningFactor: 1.0
>> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
>> > CPU:
>> > > 127200
>> > > >>>>> and CPU after applying overprovisioning: 127200
>> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 ,
>> > > Requested CPU:
>> > > >>>>> 500
>> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
>> > > >>>>> 229802639360 , Requested RAM: (1.00 GB) 1073741824
>> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
>> and
>> > > RAM
>> > > >>>>> available
>> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
>> > from
>> > > host:
>> > > >>>>> 1, used: 48200, reserved: 0, actual total: 127200, total with
>> > > >>>>> overprovisioning: 127200; requested
>> > > cpu:500,alloc_from_last_host?:false
>> > > >>>>> ,considerReservedCapacity?: true
>> > > >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
>> > from
>> > > host:
>> > > >>>>> 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total:
>> > > (250.27 GB)
>> > > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
>> > > alloc_from_last_host?:
>> > > >>>>> false , considerReservedCapacity?: true
>> > > >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable
>> host,
>> > > adding to
>> > > >>>>> list: 1
>> > > >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751
>> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator
>> > returning 4
>> > > >>>>> suitable hosts
>> > > >>>>> 2022-08-29 10:51:56,260 DEBUG [c.c.c.CapacityManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751)
>> > > >>>>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
>> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
>> state
>> > > >>>>> transited from [Starting] to [Stopped] with event
>> > [OperationFailed].
>> > > VM's
>> > > >>>>> original host: null, new host: null, host before state
>> transition:
>> > > null
>> > > >>>>> 2022-08-29 10:51:56,260 WARN [c.c.v.VirtualMachineManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751)
>> > > >>>>> (logid:ba10fec5) Unable to orchestrate start VM instance {id:
>> "83",
>> > > name:
>> > > >>>>> "v-83-VM", uuid: "b1d1f9c6-eafe-4527-9335-e649c646aab0",
>> > > >>>>> type="ConsoleProxy"} due to [Unable to create deployment, no
>> usable
>> > > volumes
>> > > >>>>> found for the VM: 83].
>> > > >>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to
>> create
>> > > >>>>> deployment, no usable volumes found for the VM: 83
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
>> > > >>>>> at
>> jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>> > > >>>>> Source)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > > >>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>> > > >>>>> at
>> > > >>>>>
>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> > > >>>>> at
>> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> > > >>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>> > > >>>>> 2022-08-29 10:51:56,262 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751)
>> > > >>>>> (logid:ba10fec5) Done executing VM work job:
>> > > >>>>>
>> > >
>> >
>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
>> > > >>>>> 2022-08-29 10:51:56,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751)
>> > > >>>>> (logid:ba10fec5) Complete async job-3539, jobStatus: FAILED,
>> > > resultCode: 0,
>> > > >>>>> result:
>> > > >>>>>
>> > >
>> >
>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODMiLCBuYW1lOiAidi04My1WTSIsIHV1aWQ6ICJiMWQxZjljNi1lYWZlLTQ1MjctOTMzNS1lNjQ5YzY0NmFhYjAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODNdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
>> > > >>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751)
>> > > >>>>> (logid:ba10fec5) Publish async job-3539 complete on message bus
>> > > >>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751)
>> > > >>>>> (logid:ba10fec5) Wake up jobs related to job-3539
>> > > >>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751)
>> > > >>>>> (logid:ba10fec5) Update db status for job-3539
>> > > >>>>> 2022-08-29 10:51:56,266 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
>> ctx-073f2751)
>> > > >>>>> (logid:ba10fec5) Wake up jobs joined with job-3539 and disjoin
>> all
>> > > subjobs
>> > > >>>>> created from job- 3539
>> > > >>>>> 2022-08-29 10:51:56,276 DEBUG [c.c.v.VmWorkJobDispatcher]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
>> > > (logid:ba10fec5) Done
>> > > >>>>> with run of VM work job: com.cloud.vm.VmWorkStart for VM 83, job
>> > > origin:
>> > > >>>>> 3445
>> > > >>>>> 2022-08-29 10:51:56,276 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
>> > > (logid:ba10fec5) Done
>> > > >>>>> executing com.cloud.vm.VmWorkStart for job-3539
>> > > >>>>> 2022-08-29 10:51:56,278 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
>> > > (logid:ba10fec5)
>> > > >>>>> Remove job-3539 from job monitoring
>> > > >>>>> 2022-08-29 10:51:56,285 WARN [c.c.c.ConsoleProxyManagerImpl]
>> > > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Unable to
>> allocate
>> > > console
>> > > >>>>> proxy standby capacity for zone [1] due to [Unable to
>> orchestrate
>> > > start VM
>> > > >>>>> instance {id: "83", name: "v-83-VM", uuid:
>> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
>> due to
>> > > [Unable
>> > > >>>>> to create deployment, no usable volumes found for the VM: 83].].
>> > > >>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to
>> > > orchestrate
>> > > >>>>> start VM instance {id: "83", name: "v-83-VM", uuid:
>> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
>> due to
>> > > [Unable
>> > > >>>>> to create deployment, no usable volumes found for the VM: 83].
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
>> > > >>>>> at
>> jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>> > > >>>>> Source)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > > >>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>> > > >>>>> at
>> > > >>>>>
>> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> > > >>>>> at
>> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> > > >>>>> at
>> > > >>>>>
>> > >
>> >
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> > > >>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>> > > >>>>> 2022-08-29 10:51:56,287 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>> > > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) received console
>> > > proxy alert
>> > > >>>>> 2022-08-29 10:51:56,288 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>> > > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Console proxy
>> > creation
>> > > >>>>> failure, zone: blabla
>> > > >>>>> 2022-08-29 10:51:56,289 WARN [c.c.a.AlertManagerImpl]
>> > > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) alertType=[10]
>> > > >>>>> dataCenterId=[1] podId=[null] clusterId=[null] message=[Console
>> > proxy
>> > > >>>>> creation failure. zone: blabla, error details: Unable to
>> > orchestrate
>> > > start
>> > > >>>>> VM instance {id: "83", name: "v-83-VM", uuid:
>> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
>> due to
>> > > [Unable
>> > > >>>>> to create deployment, no usable volumes found for the VM: 83].].
>> > > >>>>>
>> > > >>>>>
>> > > >>>>>
>> > > >>>>>
>> > > >>>>> Am Mo., 29. Aug. 2022 um 09:40 Uhr schrieb Slavka Peleva
>> > > >>>>> <sl...@storpool.com.invalid>:
>> > > >>>>>
>> > > >>>>>> Hi Chris,
>> > > >>>>>>
>> > > >>>>>> Did you recreate the system VMs? In the 4.17 version, the
>> > > >>>>>> `systemvm.iso` is
>> > > >>>>>> deprecated. In its place, you should have the file on agents
>> > > >>>>>> `/usr/share/cloudstack-common/vms/agent.zip`.
>> > > >>>>>> Can you share the complete log if the system VMs are newly
>> created
>> > > >>>>>> after
>> > > >>>>>> the upgrade?
>> > > >>>>>>
>> > > >>>>>> Best regards,
>> > > >>>>>> Slavka
>> > > >>>>>>
>> > > >>>>>> On Sun, Aug 28, 2022 at 2:16 PM Vash_X@gmx.de <Va...@gmx.de>
>> > > wrote:
>> > > >>>>>>
>> > > >>>>>> > Hi everyone,
>> > > >>>>>> >
>> > > >>>>>> > faceing some challanges again after upgradeing to 4.17.
>> > > >>>>>> > Did as explained in the docs.
>> > > >>>>>> >
>> > > >>>>>> > Afterwards I am now not able to migrate system VMs to
>> different
>> > > >>>>>> hosts.
>> > > >>>>>> > Getting the error
>> > > >>>>>> > "Exception during migrate: org.libvirt.LibvirtException:
>> Cannot
>> > > >>>>>> access
>> > > >>>>>> > storage file
>> '/usr/share/cloudstack-common/vms/systemvm.iso': No
>> > > >>>>>> such file
>> > > >>>>>> > or directory"
>> > > >>>>>> >
>> > > >>>>>> > Checked the exitstence of the file on the servers:
>> > > >>>>>> > Mangement-Server has a "systemvm.iso.bak"
>> > > >>>>>> > Hosts - no "systemvm.iso" at all present.
>> > > >>>>>> >
>> > > >>>>>> > Maybe someone has an idea on how to move on with this?
>> > > >>>>>> >
>> > > >>>>>> > Regards,
>> > > >>>>>> > Chris
>> > > >>>>>> >
>> > > >>>>>>
>> > > >>>>>
>> > >
>> >
>>
>
Re: Error migrate System VMs after upgrageing to 4.17
Posted by Slavka Peleva <sl...@storpool.com.INVALID>.
Hi Chris,
According to the logs you shared and looking at the code, I guess CS gets
the VLAN IDs by domain id, not by zone id. But to be more precise in this,
can you enable the TRACE level in your logs and share the exact query that
CS executes?
If I'm right with this, you can enable *just for now* the setting
`use.system.public.ips` (To redeploy the system VMs). This should pick
VLANs by zone ID. I hope that the deployment won't fail this time! If the
system VMs are recreated, can you disable the setting
`use.system.public.ips` (because I don't know if this will affect something
else)?
The discussion started about a problem migrating the system VMs, and
probably many other issues will appear.
Can you tell from which version you started the upgrade? (two weeks ago, I
upgraded one of my environments from 4.11.3.0 to the main branch
4.18.0.0-SNAPSHOT without any issues)
Did the upgrade was successful for the management/agents?
Did you restart the agents on all machines? My guess is the new code isn't
loaded. It's strange that CS is looking for systemvm.iso when 4.17 doesn't
know about it.
Can you share the content in the directory
`/usr/share/cloudstack-agent/lib/` from the hypervisor where the migration
failed?
Best regards,
Slavka
On Wed, Aug 31, 2022 at 2:35 AM Vash_X@gmx.de <Va...@gmx.de> wrote:
> Hi Nicolas,
>
> > What is the value for the setting
> ‘system.vm.public.ip.reservation.mode.strictness’?
> This is set to 'true'.
>
> > Can you please check if this query gets any result?
> Yes the query gives me results - all availeable "public" ip addresses of
> the reserved range for the systemVMs
>
> # public_ip_address vlan_db_id vlan_id
> 1 10.129.0.10 9 1000
> 2 10.129.0.11 9 1000
> 3 10.129.0.12 9 1000
> 4 10.129.0.13 9 1000
> 5 10.129.0.14 9 1000
> 6 10.129.0.15 9 1000
> 7 10.129.0.16 9 1000
> 8 10.129.0.18 9 1000
> 9 10.129.0.19 9 1000
> Regarding the vlanId id=[3, 10, 13, 14]
> I looked up my table "vlan" and am quite shocked:
>
> the ids 3, 13, 14 have a value for "range removed", so i don't know why the
> console proxy is looking for these networks.
> the id "10" is is still active - however the ipv4 range is not "the right
> one". That is assigned as the public ip-range for the root-account but not
> for use of systemvms...
>
> Am Di., 30. Aug. 2022 um 16:43 Uhr schrieb Nicolas Vazquez <
> Nicolas.Vazquez@shapeblue.com>:
>
> > Hi Chris,
> >
> > What is the value for the setting
> > ‘system.vm.public.ip.reservation.mode.strictness’?
> >
> > Looks like the error is not being able to get an IP address for zone
> id=1,
> > vlanId id=[3, 10, 13, 14].
> > If I understood correctly you have defined a range for system VMs? Can
> you
> > please check if this query gets any result?
> >
> > select i.public_ip_address, i.vlan_db_id, v.vlan_id from user_ip_address
> i
> > join vlan v on (i.vlan_db_id = v.id)
> > where i.forsystemvms = 1 and i.allocated is null and i.state = 'Free' and
> > v.removed is null and i.data_center_id = 1;
> >
> > Regards,
> > Nicolas Vazquez
> >
> >
> > From: Vash_X@gmx.de <Va...@gmx.de>
> > Date: Tuesday, 30 August 2022 at 09:38
> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>, Slavka
> > Peleva <sl...@storpool.com>
> > Subject: Re: Error migrate System VMs after upgrageing to 4.17
> > Hi Slavka,
> >
> > that was my first thought, too. I checked beforehand and confirmed with
> > your provided SQL Statements:
> > *`select * from user_ip_address where forsystemvms = 1;`* <---- returns
> my
> > defined (Public) IPs out of the correct range for SystemVMs
> > *`SELECT * FROM clou*d.user_ip_address where allocated IS NULL AND
> removed
> > IS NULL; <-- All IPs shown beside the one which is currently in use for
> the
> > running ssvm. Meaning from my point of view "free" for allocation for
> other
> > system VMs.
> >
> > Querys performed directly on the DB-Server.
> > I can't see a reason why there shouldn't be any IPs left for allocation.
> >
> > Also it is interesting why the "reasoning" for the failing deployment
> > changes from "no IP availeable" to "no useable volumes found".
> > How ever i can't see anything missing at the moment, what makes me even
> > more puzzeled.
> >
> > Regards
> > Chris
> >
> > Am Di., 30. Aug. 2022 um 13:31 Uhr schrieb Slavka Peleva
> > <sl...@storpool.com.invalid>:
> >
> > > Hi Chris,
> > >
> > > The deployment fails because CS cannot find a free IP address to
> allocate
> > > for the system VM. Can you check if you explicitly added a range for
> the
> > > system VMs, or you can check with this `select * from user_ip_address
> > where
> > > forsystemvms = 1;` If you don't have any ranges, you can get free IPs
> > > with `SELECT
> > > * FROM cloud.user_ip_address where allocated IS NULL AND removed IS
> > NULL;`
> > > . To get the exact query executed by CS, you need to increase the log
> > level
> > > to TRACE.
> > >
> > > Best regards,
> > > Slavka
> > >
> >
> >
> >
> > > On Mon, Aug 29, 2022 at 8:03 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
> > >
> > > > Hi Slavka,
> > > >
> > > > no, i am not using any kind of custome offerings.
> > > >
> > > > I try to provide a more detailled log this time.
> > > > Situation:
> > > > v-83-VM was the "broken" consoleproxy. So the following log starts
> with
> > > > it's expunging.
> > > > Afterwards the new consoleproxy shall be deployed - v-85-VM.
> > > >
> > > > This first deployment (logid:dcfe0999) fails due to Insufficent
> address
> > > > capacity - which i can't really understand or see in the database,
> > > checking
> > > > the availeable IPs for systemVMs (given that i might looking up the
> > wrong
> > > > tables....).
> > > > How ever CS is working as expected trying to deploy the console proxy
> > > > (logid:440fd101) Sync job-4288).
> > > > That job fails with "Unable to create deployment, no usable volumes
> > found
> > > > for the VM: 85".
> > > > Afterwards i have another try to start the consoleproxy (job-4289)
> > which
> > > > fails for the same reason. And thats how things go on in the log
> > file...
> > > >
> > > >
> > > > 2022-08-29 17:01:16,224 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> > (logid:5fe8559a)
> > > > Cleaning up hypervisor data structures (ex. SRs in XenServer) for
> > managed
> > > > storage. Data from VM instance {id: "83", name: "v-83-VM", uuid:
> > > > "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}.
> > > > 2022-08-29 17:01:16,225 DEBUG [o.a.c.e.o.VolumeOrchestrator]
> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> > (logid:5fe8559a)
> > > > Cleaning storage for vm: 83
> > > > 2022-08-29 17:01:16,233 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> > (logid:5fe8559a)
> > > > Expunged VM instance {id: "83", name: "v-83-VM", uuid:
> > > > "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
> > > > 2022-08-29 17:01:16,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> > (logid:5fe8559a)
> > > > Complete async job-4286, jobStatus: SUCCEEDED, resultCode: 0, result:
> > > >
> > >
> >
> org.apache.cloudstack.api.response.SystemVmResponse/systemvm/{"id":"b1d1f9c6-eafe-4527-9335-e649c646aab0","systemvmtype":"consoleproxy","zoneid":"48e2e928-3300-43b5-8e3a-d9b358022502","zonename":"xxxxxxxxxx","dns1":"10.129.0.4","dns2":"195.10.195.195","name":"v-83-VM","templateid":"c20799e4-a990-449a-aae6-4b67374c7aff","templatename":"systemvm-kvm-4.17.0","created":"2022-08-29T10:38:26+0200","state":"Stopped","isdynamicallyscalable":"false","hasannotations":"false"}
> > > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> > (logid:5fe8559a)
> > > > Publish async job-4286 complete on message bus
> > > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> > (logid:5fe8559a)
> > > > Wake up jobs related to job-4286
> > > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> > (logid:5fe8559a)
> > > > Update db status for job-4286
> > > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> > (logid:5fe8559a)
> > > > Wake up jobs joined with job-4286 and disjoin all subjobs created
> from
> > > job-
> > > > 4286
> > > > 2022-08-29 17:01:16,260 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286) (logid:5fe8559a) Done
> > > executing
> > > > org.apache.cloudstack.api.command.admin.systemvm.DestroySystemVmCmd
> for
> > > > job-4286
> > > > 2022-08-29 17:01:16,260 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > > > (API-Job-Executor-35:ctx-29e06f28 job-4286) (logid:5fe8559a) Remove
> > > > job-4286 from job monitoring
> > > > 2022-08-29 17:01:17,691 DEBUG [o.a.c.h.HAManagerImpl]
> > > > (BackgroundTaskPollManager-6:ctx-6af18142) (logid:fe263a70) HA health
> > > check
> > > > task is running...
> > > >
> > > > 2022-08-29 17:01:21,694 DEBUG [o.a.c.h.HAManagerImpl]
> > > > (BackgroundTaskPollManager-6:ctx-89b1442f) (logid:0efc40b2) HA health
> > > check
> > > > task is running...
> > > > 2022-08-29 17:01:25,698 DEBUG [o.a.c.h.HAManagerImpl]
> > > > (BackgroundTaskPollManager-5:ctx-62313d05) (logid:7073663f) HA health
> > > check
> > > > task is running...
> > > > 2022-08-29 17:01:25,767 DEBUG [c.c.s.StatsCollector]
> > > > (secstorage-1:ctx-c80e5a6e) (logid:f8e3eb60) Verifying image storage
> > [1].
> > > > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
> > > > 2022-08-29 17:01:25,773 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > > > (secstorage-1:ctx-c80e5a6e) (logid:f8e3eb60) Zone [1] is ready to
> > launch
> > > > secondary storage VM.
> > > > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Zone 1 is ready to
> > launch
> > > > console proxy
> > > > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Expand console proxy
> > > standby
> > > > capacity for zone xxxxxxxxxx
> > > > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating console
> proxy
> > > > standby capacity for zone [1].
> > > > 2022-08-29 17:01:26,032 INFO [c.c.c.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) No stopped console
> proxy
> > > is
> > > > available, need to allocate a new console proxy
> > > > 2022-08-29 17:01:26,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Assign console proxy
> > from
> > > a
> > > > newly started instance for request from data center : 1
> > > > 2022-08-29 17:01:26,056 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing network
> > > > configuration for offering [Network Offering
> > > > [1-Public-System-Public-Network]: Network {"id": 200, "name": "null",
> > > > "uuid": "1b0bf965-a391-478d-833d-3e37e4a15eb8", "networkofferingid":
> 1}
> > > > 2022-08-29 17:01:26,056 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
> > > > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account {"id":
> 1,
> > > > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
> > > > 2022-08-29 17:01:26,064 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing network
> > > > configuration for offering [Network Offering
> > > > [3-Control-System-Control-Network]: Network {"id": 202, "name":
> "null",
> > > > "uuid": "2af42518-fbf4-4a35-9d6b-370e6ad315c7", "networkofferingid":
> 3}
> > > > 2022-08-29 17:01:26,064 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
> > > > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account {"id":
> 1,
> > > > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
> > > > 2022-08-29 17:01:26,069 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing network
> > > > configuration for offering [Network Offering
> > > > [2-Management-System-Management-Network]: Network {"id": 201, "name":
> > > > "null", "uuid": "53711d87-a3c6-4f33-bfce-c30e0de72760",
> > > > "networkofferingid": 2}
> > > > 2022-08-29 17:01:26,069 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
> > > > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account {"id":
> 1,
> > > > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
> > > > 2022-08-29 17:01:26,077 INFO [c.c.v.VirtualMachineManagerImpl]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) allocating virtual
> > machine
> > > > from template:c20799e4-a990-449a-aae6-4b67374c7aff with
> > hostname:v-85-VM
> > > > and 3 networks
> > > > 2022-08-29 17:01:26,079 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating entries for
> > VM:
> > > > VM instance {id: "85", name: "v-85-VM", uuid:
> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
> > > > 2022-08-29 17:01:26,082 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating nics for VM
> > > > instance {id: "85", name: "v-85-VM", uuid:
> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
> > > > 2022-08-29 17:01:26,083 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating nic for vm
> VM
> > > > instance {id: "85", name: "v-85-VM", uuid:
> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} in
> network
> > > > Network {"id": 200, "name": "null", "uuid":
> > > > "1b0bf965-a391-478d-833d-3e37e4a15eb8", "networkofferingid": 1} with
> > > > requested profile NicProfile
> > > >
> > >
> >
> {"broadcastUri":null,"iPv4Address":null,"id":0,"reservationId":null,"vmId":0}
> > > > 2022-08-29 17:01:26,089 WARN [c.c.n.IpAddressManagerImpl]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to get ip
> address
> > > in
> > > > zone id=1, vlanId id=[3, 10, 13, 14]
> > > > 2022-08-29 17:01:26,090 DEBUG [c.c.u.d.T.Transaction]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Rolling back the
> > > > transaction: Time = 8 Name = consoleproxy-1; called by
> > > >
> > >
> >
> -TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-Transaction.execute:38-IpAddressManagerImpl.listAvailablePublicIps:824-IpAddressManagerImpl.fetchNewPublicIp:811-IpAddressManagerImpl.fetchNewPublicIp:803-IpAddressManagerImpl.assignPublicIpAddress:784-PublicNetworkGuru.getIp:125-PublicNetworkGuru.allocate:172-NetworkOrchestrator.allocateNic:973-NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice:823
> > > > 2022-08-29 17:01:26,090 WARN [c.c.c.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to allocate
> proxy
> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone
> > [1]
> > > > due to [Insufficient address capacity].
> > > > com.cloud.exception.InsufficientAddressCapacityException:
> Insufficient
> > > > address capacityScope=interface com.cloud.dc.DataCenter; id=1
> > > > at
> > > >
> > >
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
> > > > at
> > > >
> > >
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> > > > at
> > > >
> > >
> >
> com.cloud.network.IpAddressManagerImpl.listAvailablePublicIps(IpAddressManagerImpl.java:824)
> > > > at
> > > >
> > >
> >
> com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl.java:811)
> > > > at
> > > >
> > >
> >
> com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl.java:803)
> > > > at
> > > >
> > >
> >
> com.cloud.network.IpAddressManagerImpl.assignPublicIpAddress(IpAddressManagerImpl.java:784)
> > > > at
> > > >
> > >
> >
> com.cloud.network.guru.PublicNetworkGuru.getIp(PublicNetworkGuru.java:125)
> > > > at
> > > >
> > >
> >
> com.cloud.network.guru.PublicNetworkGuru.allocate(PublicNetworkGuru.java:172)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocateNic(NetworkOrchestrator.java:973)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice(NetworkOrchestrator.java:823)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$3.doInTransactionWithoutResult(NetworkOrchestrator.java:797)
> > > > at
> > > >
> > >
> >
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
> > > > at
> > > >
> > >
> >
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocate(NetworkOrchestrator.java:778)
> > > > at
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl$1.doInTransactionWithoutResult(VirtualMachineManagerImpl.java:478)
> > > > at
> > > >
> > >
> >
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
> > > > at
> > > >
> > >
> >
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> > > > at
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:469)
> > > > at
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:528)
> > > > at
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:700)
> > > > at
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
> > > > at
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
> > > > at
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
> > > > at
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
> > > > at
> > > com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
> > > > at
> > > >
> > com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
> > > > at
> > > >
> > >
> >
> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > > at java.base/java.lang.Thread.run(Thread.java:829)
> > > > 2022-08-29 17:01:26,091 WARN [c.c.c.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to allocate
> > console
> > > > proxy standby capacity for zone [1] due to [Unable to allocate proxy
> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone
> > [1]
> > > > due to [Insufficient address capacity].].
> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to allocate
> > proxy
> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone
> > [1]
> > > > due to [Insufficient address capacity].
> > > > at
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:704)
> > > > at
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
> > > > at
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
> > > > at
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
> > > > at
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
> > > > at
> > > com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
> > > > at
> > > >
> > com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
> > > > at
> > > >
> > >
> >
> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > > at java.base/java.lang.Thread.run(Thread.java:829)
> > > > Caused by: com.cloud.exception.InsufficientAddressCapacityException:
> > > > Insufficient address capacityScope=interface com.cloud.dc.DataCenter;
> > > id=1
> > > > at
> > > >
> > >
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
> > > > at
> > > >
> > >
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> > > > ... 38 more
> > > > 2022-08-29 17:01:26,091 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) received console proxy
> > > alert
> > > > 2022-08-29 17:01:26,092 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Console proxy creation
> > > > failure, zone: xxxxxxxxxx
> > > > 2022-08-29 17:01:26,093 WARN [c.c.a.AlertManagerImpl]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) alertType=[10]
> > > > dataCenterId=[1] podId=[null] clusterId=[null] message=[Console proxy
> > > > creation failure. zone: xxxxxxxxxx, error details: Unable to allocate
> > > proxy
> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone
> > [1]
> > > > due to [Insufficient address capacity].].
> > > > 2022-08-29 17:01:26,096 WARN [c.c.a.AlertManagerImpl]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) No recipients set in
> > > global
> > > > setting 'alert.email.addresses', skipping sending alert with subject
> > > > [Console proxy creation failure. zone: xxxxxxxxxx, error details:
> > Unable
> > > to
> > > > allocate proxy [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone
> > [1]
> > > > due to [Insufficient address capacity].] and content [Console proxy
> > > > creation failure (zone xxxxxxxxxx)].
> > > > 2022-08-29 17:01:26,096 WARN [c.c.v.SystemVmLoadScanner]
> > > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unexpected exception
> > > Unable
> > > > to allocate proxy [Console VM instance {id: "85", name: "v-85-VM",
> > uuid:
> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone
> > [1]
> > > > due to [Insufficient address capacity].
> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to allocate
> > proxy
> > > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone
> > [1]
> > > > due to [Insufficient address capacity].
> > > > at
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:704)
> > > > at
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
> > > > at
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
> > > > at
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
> > > > at
> > > >
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
> > > > at
> > > com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
> > > > at
> > > >
> > com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
> > > > at
> > > >
> > >
> >
> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > > at java.base/java.lang.Thread.run(Thread.java:829)
> > > > Caused by: com.cloud.exception.InsufficientAddressCapacityException:
> > > > Insufficient address capacityScope=interface com.cloud.dc.DataCenter;
> > > id=1
> > > > at
> > > >
> > >
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
> > > > at
> > > >
> > >
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
> > > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> > > > ... 38 more
> > > > 2022-08-29 17:01:29,700 DEBUG [o.a.c.h.HAManagerImpl]
> > > > (BackgroundTaskPollManager-5:ctx-994465bd) (logid:1e2822fd) HA health
> > > check
> > > > task is running...
> > > > 2022-08-29 17:01:30,444 DEBUG
> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > > > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 2 routers
> > to
> > > > update status.
> > > > 2022-08-29 17:01:30,449 DEBUG
> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > > > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 1 VPC's
> to
> > > > update Redundant State.
> > > > 2022-08-29 17:01:30,451 DEBUG
> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > > > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 0
> networks
> > to
> > > > update RvR status.
> > > > 2022-08-29 17:01:30,459 DEBUG [c.c.a.t.Request]
> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
> > > > 6-4833769775052102463: Sending { Cmd , MgmtId: 90520740759984, via:
> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> > > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
> > > router.name
> > >
> >
> ":"r-79-VM","router.ip":"169.254.144.198"},"wait":"30","bypassHostMaintenance":"false"}}]
> > > > }
> > > > 2022-08-29 17:01:30,643 DEBUG [c.c.a.t.Request]
> > > > (AgentManager-Handler-12:null) (logid:) Seq 6-4833769775052102463:
> > > > Processing: { Ans: , MgmtId: 90520740759984, via: 6, Ver: v1, Flags:
> > 10,
> > > >
> > >
> >
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"PRIMARY","result":"true","details":"Status:
> > > > PRIMARY
> > > > ","wait":"0","bypassHostMaintenance":"false"}}] }
> > > > 2022-08-29 17:01:30,643 DEBUG [c.c.a.t.Request]
> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
> > > > 6-4833769775052102463: Received: { Ans: , MgmtId: 90520740759984,
> via:
> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer
> } }
> > > > 2022-08-29 17:01:30,643 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08)
> Details
> > > from
> > > > executing class com.cloud.agent.api.CheckRouterCommand: Status:
> PRIMARY
> > > >
> > > > 2022-08-29 17:01:30,655 DEBUG [c.c.a.t.Request]
> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
> > > > 1-5410793477308947401: Sending { Cmd , MgmtId: 90520740759984, via:
> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> > > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
> > > router.name
> > >
> >
> ":"r-80-VM","router.ip":"169.254.217.200"},"wait":"30","bypassHostMaintenance":"false"}}]
> > > > }
> > > > 2022-08-29 17:01:30,796 DEBUG [c.c.a.t.Request]
> > > > (AgentManager-Handler-7:null) (logid:) Seq 1-5410793477308947401:
> > > > Processing: { Ans: , MgmtId: 90520740759984, via: 1, Ver: v1, Flags:
> > 10,
> > > >
> > >
> >
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"BACKUP","result":"true","details":"Status:
> > > > BACKUP
> > > > ","wait":"0","bypassHostMaintenance":"false"}}] }
> > > > 2022-08-29 17:01:30,796 DEBUG [c.c.a.t.Request]
> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
> > > > 1-5410793477308947401: Received: { Ans: , MgmtId: 90520740759984,
> via:
> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer
> } }
> > > > 2022-08-29 17:01:30,796 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08)
> Details
> > > from
> > > > executing class com.cloud.agent.api.CheckRouterCommand: Status:
> BACKUP
> > > >
> > > > 2022-08-29 17:01:33,703 DEBUG [o.a.c.h.HAManagerImpl]
> > > > (BackgroundTaskPollManager-5:ctx-70bca202) (logid:b332dbe0) HA health
> > > check
> > > > task is running...
> > > > 2022-08-29 17:01:34,083 DEBUG [c.c.s.StatsCollector]
> > > > (StatsCollector-6:ctx-213e2ecc) (logid:aa0584c9) AutoScaling Monitor
> is
> > > > running...
> > > > 2022-08-29 17:01:37,707 DEBUG [o.a.c.h.HAManagerImpl]
> > > > (BackgroundTaskPollManager-6:ctx-57493517) (logid:5093e75f) HA health
> > > check
> > > > task is running...
> > > > 2022-08-29 17:01:38,640 DEBUG [c.c.s.StatsCollector]
> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) HostStatsCollector
> is
> > > > running...
> > > > 2022-08-29 17:01:38,655 DEBUG [c.c.a.t.Request]
> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
> > > 1-5410793477308947402:
> > > > Received: { Ans: , MgmtId: 90520740759984, via:
> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer
> }
> > }
> > > > 2022-08-29 17:01:38,707 DEBUG [c.c.a.t.Request]
> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
> > > 5-8959067033723473395:
> > > > Received: { Ans: , MgmtId: 90520740759984, via:
> > > > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer
> }
> > }
> > > > 2022-08-29 17:01:38,717 DEBUG [c.c.a.t.Request]
> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
> > > 6-4833769775052102464:
> > > > Received: { Ans: , MgmtId: 90520740759984, via:
> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer
> }
> > }
> > > > 2022-08-29 17:01:38,725 DEBUG [c.c.a.t.Request]
> > > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
> > > 7-295548725546192472:
> > > > Received: { Ans: , MgmtId: 90520740759984, via:
> > > > 7(srv-5-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer
> }
> > }
> > > > 2022-08-29 17:01:41,709 DEBUG [o.a.c.h.HAManagerImpl]
> > > > (BackgroundTaskPollManager-5:ctx-a65ae462) (logid:bc2b7320) HA health
> > > check
> > > > task is running...
> > > > 2022-08-29 17:01:45,712 DEBUG [o.a.c.h.HAManagerImpl]
> > > > (BackgroundTaskPollManager-2:ctx-05491ace) (logid:b2232c4d) HA health
> > > check
> > > > task is running...
> > > > 2022-08-29 17:01:47,019 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > (AgentManager-Handler-14:null) (logid:) Ping from Routing host
> > > > 7(srv-5-comp.aaaaaaaa.bbb)
> > > >
> > > > 2022-08-29 17:01:49,715 DEBUG [o.a.c.h.HAManagerImpl]
> > > > (BackgroundTaskPollManager-2:ctx-7943581c) (logid:504c98ff) HA health
> > > check
> > > > task is running...
> > > > 2022-08-29 17:01:51,881 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > (AgentManager-Handler-13:null) (logid:) Ping from Routing host
> > > > 1(srv-2-comp.aaaaaaaa.bbb)
> > > >
> > > > 2022-08-29 17:01:53,718 DEBUG [o.a.c.h.HAManagerImpl]
> > > > (BackgroundTaskPollManager-4:ctx-b11cfa22) (logid:3163613f) HA health
> > > check
> > > > task is running...
> > > > 2022-08-29 17:01:53,955 DEBUG [c.c.a.t.Request]
> > > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
> > > 1-5410793477308947403:
> > > > Received: { Ans: , MgmtId: 90520740759984, via:
> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer }
> }
> > > > 2022-08-29 17:01:54,025 DEBUG [c.c.a.t.Request]
> > > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
> > > 5-8959067033723473396:
> > > > Received: { Ans: , MgmtId: 90520740759984, via:
> > > > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer }
> }
> > > > 2022-08-29 17:01:54,096 DEBUG [c.c.a.t.Request]
> > > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
> > > 6-4833769775052102465:
> > > > Received: { Ans: , MgmtId: 90520740759984, via:
> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer }
> }
> > > > 2022-08-29 17:01:54,217 DEBUG [c.c.a.t.Request]
> > > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
> > > 7-295548725546192473:
> > > > Received: { Ans: , MgmtId: 90520740759984, via:
> > > > 7(srv-5-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer }
> }
> > > > 2022-08-29 17:01:55,765 DEBUG [c.c.s.StatsCollector]
> > > > (secstorage-1:ctx-7d9995d7) (logid:e388a8b8) Verifying image storage
> > [1].
> > > > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
> > > > 2022-08-29 17:01:55,771 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > > > (secstorage-1:ctx-7d9995d7) (logid:e388a8b8) Zone [1] is ready to
> > launch
> > > > secondary storage VM.
> > > > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Zone 1 is ready to
> > launch
> > > > console proxy
> > > > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Expand console proxy
> > > standby
> > > > capacity for zone xxxxxxxxxx
> > > > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Allocating console
> proxy
> > > > standby capacity for zone [1].
> > > > 2022-08-29 17:01:56,034 INFO [c.c.c.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Found a stopped
> console
> > > > proxy, starting it. Vm id : 85
> > > > 2022-08-29 17:01:56,035 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) start parameter value
> of
> > > > enterHardwareSetup == <very null> during processing of queued job
> > > > 2022-08-29 17:01:56,044 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Sync job-4288
> execution
> > on
> > > > object VmWorkJobQueue.85
> > > > 2022-08-29 17:01:56,173 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (AsyncJobMgr-Heartbeat-1:ctx-e75826b1) (logid:e80fe659) Execute
> > > sync-queue
> > > > item: SyncQueueItemVO {id:2105, queueId: 2105, contentType: AsyncJob,
> > > > contentId: 4288, lastProcessMsid: 90520740759984, lastprocessNumber:
> 1,
> > > > lastProcessTime: Mon Aug 29 17:01:56 CEST 2022, created: Mon Aug 29
> > > > 17:01:56 CEST 2022}
> > > > 2022-08-29 17:01:56,174 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (AsyncJobMgr-Heartbeat-1:ctx-e75826b1) (logid:e80fe659) Schedule
> queued
> > > > job-4288
> > > > 2022-08-29 17:01:56,179 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
> (logid:7e671dd0)
> > > Add
> > > > job-4288 into job monitoring
> > > > 2022-08-29 17:01:56,186 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
> (logid:ba10fec5)
> > > > Executing AsyncJobVO: {id:4288, userId: 1, accountId: 1,
> instanceType:
> > > > null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
> > > >
> > >
> >
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAVXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
> > > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> > > > result: null, initMsid: 90520740759984, completeMsid: null,
> > lastUpdated:
> > > > null, lastPolled: null, created: Mon Aug 29 17:01:56 CEST 2022,
> > removed:
> > > > null}
> > > > 2022-08-29 17:01:56,186 DEBUG [c.c.v.VmWorkJobDispatcher]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
> (logid:ba10fec5)
> > > Run
> > > > VM work job: com.cloud.vm.VmWorkStart for VM 85, job origin: 3445
> > > > 2022-08-29 17:01:56,187 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > > (logid:ba10fec5) Execute VM work job:
> > > >
> > >
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> > > > 2022-08-29 17:01:56,189 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > > (logid:ba10fec5) orchestrating VM start for 'v-85-VM'
> > > > com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
> > > > 2022-08-29 17:01:56,196 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
> > > > transited from [Stopped] to [Starting] with event [StartRequested].
> > VM's
> > > > original host: null, new host: null, host before state transition:
> null
> > > > 2022-08-29 17:01:56,196 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > > (logid:ba10fec5) Successfully transitioned to start state for VM
> > instance
> > > > {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80",
> > > > type="ConsoleProxy"} reservation id =
> > > 0ab68202-d8be-469c-9b2a-87a55c2dc489
> > > > 2022-08-29 17:01:56,200 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > > (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts:
> null
> > > > 2022-08-29 17:01:56,202 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > > (logid:ba10fec5) VM start attempt #1
> > > > 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > > (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
> > > > 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > > (logid:ba10fec5) Trying to allocate a host and storage pools from
> dc:1,
> > > > pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
> > > > 1073741824
> > > > 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > > (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
> > > > 2022-08-29 17:01:56,212 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > > (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
> > > > 2022-08-29 17:01:56,214 DEBUG [c.c.d.FirstFitPlanner]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > > (logid:ba10fec5) Searching all possible resources under this Zone: 1
> > > > 2022-08-29 17:01:56,215 DEBUG [c.c.d.FirstFitPlanner]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > > (logid:ba10fec5) Listing clusters in order of aggregate capacity,
> that
> > > have
> > > > (at least one host with) enough CPU and RAM capacity under this
> Zone: 1
> > > > 2022-08-29 17:01:56,218 DEBUG [c.c.d.FirstFitPlanner]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > > (logid:ba10fec5) Removing from the clusterId list these clusters from
> > > avoid
> > > > set: []
> > > > 2022-08-29 17:01:56,224 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > > (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
> > > > 2022-08-29 17:01:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested
> with
> > > > Custom[UEFI] Boot Type false
> > > > 2022-08-29 17:01:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in dc: 1
> > > > pod:1 cluster:1
> > > > 2022-08-29 17:01:56,226 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4
> > hosts
> > > to
> > > > check for allocation: [Host {"id": "5", "name":
> > > "srv-4-comp.aaaaaaaa.bbb",
> > > > "uuid": "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"},
> Host
> > > > {"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb", "uuid":
> > > > "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
> {"id":
> > > "6",
> > > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> > > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
> {"id":
> > > "1",
> > > > "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> > > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> > > > 2022-08-29 17:01:56,232 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for
> allocation
> > > > after prioritization: [Host {"id": "5", "name":
> > > "srv-4-comp.aaaaaaaa.bbb",
> > > > "uuid": "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"},
> Host
> > > > {"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb", "uuid":
> > > > "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
> {"id":
> > > "6",
> > > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> > > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
> {"id":
> > > "1",
> > > > "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> > > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> > > > 2022-08-29 17:01:56,232 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for speed=500Mhz,
> > > > Ram=1024 MB
> > > > 2022-08-29 17:01:56,232 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
> > > > srv-4-comp.aaaaaaaa.bbb, uuid: 134c625d-15ab-461a-a38d-36fb0b979b5d}
> is
> > > KVM
> > > > hypervisor type, no max guest limit check needed
> > > > 2022-08-29 17:01:56,234 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu capability
> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed:
> > 500
> > > > 2022-08-29 17:01:56,234 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has
> > enough
> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> > 1073741824 ,
> > > > cpuOverprovisioningFactor: 1.0
> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> > > 127200
> > > > and CPU after applying overprovisioning: 127200
> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 ,
> Requested
> > > > CPU: 500
> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
> > > > 251009040384 , Requested RAM: (1.00 GB) 1073741824
> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
> RAM
> > > > available
> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> > > host:
> > > > 5, used: 11100, reserved: 0, actual total: 127200, total with
> > > > overprovisioning: 127200; requested
> cpu:500,alloc_from_last_host?:false
> > > > ,considerReservedCapacity?: true
> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> > > host:
> > > > 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total:
> (250.27
> > > GB)
> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
> > alloc_from_last_host?:
> > > > false , considerReservedCapacity?: true
> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> > adding
> > > to
> > > > list: 5
> > > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
> > > > srv-5-comp.aaaaaaaa.bbb, uuid: 8c753ce1-0253-49c9-9c7f-1f971a719ce6}
> is
> > > KVM
> > > > hypervisor type, no max guest limit check needed
> > > > 2022-08-29 17:01:56,238 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu capability
> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed:
> > 500
> > > > 2022-08-29 17:01:56,238 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has
> > enough
> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> > 1073741824 ,
> > > > cpuOverprovisioningFactor: 1.0
> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> > > 127200
> > > > and CPU after applying overprovisioning: 127200
> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 ,
> Requested
> > > CPU:
> > > > 500
> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
> > > > 189373743104 , Requested RAM: (1.00 GB) 1073741824
> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
> RAM
> > > > available
> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> > > host:
> > > > 7, used: 90100, reserved: 0, actual total: 127200, total with
> > > > overprovisioning: 127200; requested
> cpu:500,alloc_from_last_host?:false
> > > > ,considerReservedCapacity?: true
> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> > > host:
> > > > 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total:
> (250.27
> > > GB)
> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
> > alloc_from_last_host?:
> > > > false , considerReservedCapacity?: true
> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> > adding
> > > to
> > > > list: 7
> > > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
> > > > srv-3-comp.aaaaaaaa.bbb, uuid: 6858713b-7e17-46d8-854d-24a13f9c7001}
> is
> > > KVM
> > > > hypervisor type, no max guest limit check needed
> > > > 2022-08-29 17:01:56,242 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu capability
> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed:
> > 500
> > > > 2022-08-29 17:01:56,242 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has
> > enough
> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> > 1073741824 ,
> > > > cpuOverprovisioningFactor: 1.0
> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> > > 127200
> > > > and CPU after applying overprovisioning: 127200
> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 ,
> Requested
> > > CPU:
> > > > 500
> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
> > > > 204032835584 , Requested RAM: (1.00 GB) 1073741824
> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
> RAM
> > > > available
> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> > > host:
> > > > 6, used: 42900, reserved: 0, actual total: 127200, total with
> > > > overprovisioning: 127200; requested
> cpu:500,alloc_from_last_host?:false
> > > > ,considerReservedCapacity?: true
> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> > > host:
> > > > 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total:
> (250.27
> > > GB)
> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
> > alloc_from_last_host?:
> > > > false , considerReservedCapacity?: true
> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> > adding
> > > to
> > > > list: 6
> > > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
> > > > srv-2-comp.aaaaaaaa.bbb, uuid: 883f7f62-55f2-42f4-8242-16b6847c4204}
> is
> > > KVM
> > > > hypervisor type, no max guest limit check needed
> > > > 2022-08-29 17:01:56,246 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu capability
> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed:
> > 500
> > > > 2022-08-29 17:01:56,246 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has
> > enough
> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> > 1073741824 ,
> > > > cpuOverprovisioningFactor: 1.0
> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> > > 127200
> > > > and CPU after applying overprovisioning: 127200
> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 ,
> Requested
> > > CPU:
> > > > 500
> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
> > > > 229802639360 , Requested RAM: (1.00 GB) 1073741824
> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
> RAM
> > > > available
> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> > > host:
> > > > 1, used: 48200, reserved: 0, actual total: 127200, total with
> > > > overprovisioning: 127200; requested
> cpu:500,alloc_from_last_host?:false
> > > > ,considerReservedCapacity?: true
> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> > > host:
> > > > 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total:
> (250.27
> > > GB)
> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
> > alloc_from_last_host?:
> > > > false , considerReservedCapacity?: true
> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> > adding
> > > to
> > > > list: 1
> > > > 2022-08-29 17:01:56,247 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator returning 4
> > > > suitable hosts
> > > > 2022-08-29 17:01:56,258 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
> > > > transited from [Starting] to [Stopped] with event [OperationFailed].
> > VM's
> > > > original host: null, new host: null, host before state transition:
> null
> > > > 2022-08-29 17:01:56,258 WARN [c.c.v.VirtualMachineManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > > (logid:ba10fec5) Unable to orchestrate start VM instance {id: "85",
> > name:
> > > > "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
> > > > type="ConsoleProxy"} due to [Unable to create deployment, no usable
> > > volumes
> > > > found for the VM: 85].
> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to create
> > > > deployment, no usable volumes found for the VM: 85
> > > > at
> > > >
> > >
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
> > > > at
> > > >
> > >
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
> > > > at
> > > >
> > >
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
> > > > at
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
> > > > at
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> > Source)
> > > > at
> > > >
> > >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > > > at
> > > >
> > >
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > > > at
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > > > at
> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > > > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > > at java.base/java.lang.Thread.run(Thread.java:829)
> > > > 2022-08-29 17:01:56,259 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > > (logid:ba10fec5) Done executing VM work job:
> > > >
> > >
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> > > > 2022-08-29 17:01:56,260 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > > (logid:ba10fec5) Complete async job-4288, jobStatus: FAILED,
> > resultCode:
> > > 0,
> > > > result:
> > > >
> > >
> >
> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODUiLCBuYW1lOiAidi04NS1WTSIsIHV1aWQ6ICIxMTBkODYyMC1jMDI5LTRjYTktOTA4NC0yMTAwOTk3MjVhODAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODVdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
> > > > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > > (logid:ba10fec5) Publish async job-4288 complete on message bus
> > > > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > > (logid:ba10fec5) Wake up jobs related to job-4288
> > > > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > > (logid:ba10fec5) Update db status for job-4288
> > > > 2022-08-29 17:01:56,262 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > > (logid:ba10fec5) Wake up jobs joined with job-4288 and disjoin all
> > > subjobs
> > > > created from job- 4288
> > > > 2022-08-29 17:01:56,270 DEBUG [c.c.v.VmWorkJobDispatcher]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
> (logid:ba10fec5)
> > > > Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 85, job
> > > > origin: 3445
> > > > 2022-08-29 17:01:56,270 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
> (logid:ba10fec5)
> > > > Done executing com.cloud.vm.VmWorkStart for job-4288
> > > > 2022-08-29 17:01:56,272 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288)
> (logid:ba10fec5)
> > > > Remove job-4288 from job monitoring
> > > > 2022-08-29 17:01:56,277 WARN [c.c.c.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Unable to allocate
> > console
> > > > proxy standby capacity for zone [1] due to [Unable to orchestrate
> start
> > > VM
> > > > instance {id: "85", name: "v-85-VM", uuid:
> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> > > [Unable
> > > > to create deployment, no usable volumes found for the VM: 85].].
> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to
> orchestrate
> > > > start VM instance {id: "85", name: "v-85-VM", uuid:
> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> > > [Unable
> > > > to create deployment, no usable volumes found for the VM: 85].
> > > > at
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> > Source)
> > > > at
> > > >
> > >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > > > at
> > > >
> > >
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > > > at
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > > > at
> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > > > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > > at java.base/java.lang.Thread.run(Thread.java:829)
> > > > 2022-08-29 17:01:56,277 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) received console proxy
> > > alert
> > > > 2022-08-29 17:01:56,278 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Console proxy creation
> > > > failure, zone: xxxxxxxxxx
> > > > 2022-08-29 17:01:56,279 WARN [c.c.a.AlertManagerImpl]
> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) alertType=[10]
> > > > dataCenterId=[1] podId=[null] clusterId=[null] message=[Console proxy
> > > > creation failure. zone: xxxxxxxxxx, error details: Unable to
> > orchestrate
> > > > start VM instance {id: "85", name: "v-85-VM", uuid:
> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> > > [Unable
> > > > to create deployment, no usable volumes found for the VM: 85].].
> > > > 2022-08-29 17:01:56,284 WARN [c.c.a.AlertManagerImpl]
> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) No recipients set in
> > > global
> > > > setting 'alert.email.addresses', skipping sending alert with subject
> > > > [Console proxy creation failure. zone: xxxxxxxxxx, error details:
> > Unable
> > > to
> > > > orchestrate start VM instance {id: "85", name: "v-85-VM", uuid:
> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> > > [Unable
> > > > to create deployment, no usable volumes found for the VM: 85].] and
> > > content
> > > > [Console proxy creation failure (zone xxxxxxxxxx)].
> > > > 2022-08-29 17:01:56,284 WARN [c.c.v.SystemVmLoadScanner]
> > > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Unexpected exception
> > > Unable
> > > > to orchestrate start VM instance {id: "85", name: "v-85-VM", uuid:
> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> > > [Unable
> > > > to create deployment, no usable volumes found for the VM: 85].
> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to
> orchestrate
> > > > start VM instance {id: "85", name: "v-85-VM", uuid:
> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> > > [Unable
> > > > to create deployment, no usable volumes found for the VM: 85].
> > > > at
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> > Source)
> > > > at
> > > >
> > >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > > > at
> > > >
> > >
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > > > at
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > > > at
> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > > > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > > at java.base/java.lang.Thread.run(Thread.java:829)
> > > > 2022-08-29 17:01:57,721 DEBUG [o.a.c.h.HAManagerImpl]
> > > > (BackgroundTaskPollManager-4:ctx-9528226f) (logid:4512ff9b) HA health
> > > check
> > > > task is running...
> > > > 2022-08-29 17:01:59,504 DEBUG [c.c.s.StatsCollector]
> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) StorageCollector is
> > > > running...
> > > > 2022-08-29 17:01:59,518 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
> > > getCommandHostDelegation:
> > > > class com.cloud.agent.api.GetStorageStatsCommand
> > > > 2022-08-29 17:01:59,518 DEBUG [c.c.h.XenServerGuru]
> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning the
> > > > default host to execute commands because the command is not of Copy
> > type.
> > > > 2022-08-29 17:01:59,571 DEBUG [c.c.a.t.Request]
> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
> > > > 11-7523826127475836583: Received: { Ans: , MgmtId: 90520740759984,
> > via:
> > > > 11(s-32-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> > > > 2022-08-29 17:01:59,574 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
> > > getCommandHostDelegation:
> > > > class com.cloud.agent.api.GetStorageStatsCommand
> > > > 2022-08-29 17:01:59,574 DEBUG [c.c.h.XenServerGuru]
> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning the
> > > > default host to execute commands because the command is not of Copy
> > type.
> > > > 2022-08-29 17:01:59,615 DEBUG [c.c.a.t.Request]
> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
> > > 5-8959067033723473397:
> > > > Received: { Ans: , MgmtId: 90520740759984, via:
> > > > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
> GetStorageStatsAnswer
> > > } }
> > > > 2022-08-29 17:01:59,624 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
> > > getCommandHostDelegation:
> > > > class com.cloud.agent.api.GetStorageStatsCommand
> > > > 2022-08-29 17:01:59,624 DEBUG [c.c.h.XenServerGuru]
> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning the
> > > > default host to execute commands because the command is not of Copy
> > type.
> > > > 2022-08-29 17:01:59,653 DEBUG [c.c.a.t.Request]
> > > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
> > > 6-4833769775052102466:
> > > > Received: { Ans: , MgmtId: 90520740759984, via:
> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, {
> GetStorageStatsAnswer
> > > } }
> > > > 2022-08-29 17:01:59,979 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > (AgentManager-Handler-12:null) (logid:) Ping from Routing host
> > > > 5(srv-4-comp.aaaaaaaa.bbb)
> > > >
> > > > 2022-08-29 17:02:00,269 DEBUG [c.c.s.StatsCollector]
> > > > (StatsCollector-5:ctx-d2b6ffd3) (logid:95201678) DbCollector is
> > > running...
> > > > 2022-08-29 17:02:00,443 DEBUG
> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > > > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 2 routers
> > to
> > > > update status.
> > > > 2022-08-29 17:02:00,449 DEBUG
> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > > > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 1 VPC's
> to
> > > > update Redundant State.
> > > > 2022-08-29 17:02:00,450 DEBUG
> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > > > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 0
> networks
> > to
> > > > update RvR status.
> > > > 2022-08-29 17:02:00,457 DEBUG [c.c.a.t.Request]
> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
> > > > 6-4833769775052102467: Sending { Cmd , MgmtId: 90520740759984, via:
> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> > > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
> > > router.name
> > >
> >
> ":"r-79-VM","router.ip":"169.254.144.198"},"wait":"30","bypassHostMaintenance":"false"}}]
> > > > }
> > > > 2022-08-29 17:02:00,599 DEBUG [c.c.a.t.Request]
> > > > (AgentManager-Handler-7:null) (logid:) Seq 6-4833769775052102467:
> > > > Processing: { Ans: , MgmtId: 90520740759984, via: 6, Ver: v1, Flags:
> > 10,
> > > >
> > >
> >
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"PRIMARY","result":"true","details":"Status:
> > > > PRIMARY
> > > > ","wait":"0","bypassHostMaintenance":"false"}}] }
> > > > 2022-08-29 17:02:00,599 DEBUG [c.c.a.t.Request]
> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
> > > > 6-4833769775052102467: Received: { Ans: , MgmtId: 90520740759984,
> via:
> > > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer
> } }
> > > > 2022-08-29 17:02:00,599 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f)
> Details
> > > > from executing class com.cloud.agent.api.CheckRouterCommand: Status:
> > > PRIMARY
> > > >
> > > > 2022-08-29 17:02:00,612 DEBUG [c.c.a.t.Request]
> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
> > > > 1-5410793477308947404: Sending { Cmd , MgmtId: 90520740759984, via:
> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> > > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
> > > router.name
> > >
> >
> ":"r-80-VM","router.ip":"169.254.217.200"},"wait":"30","bypassHostMaintenance":"false"}}]
> > > > }
> > > > 2022-08-29 17:02:00,748 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > (AgentManager-Handler-4:null) (logid:) Ping from 11(s-32-VM)
> > > > 2022-08-29 17:02:00,750 DEBUG [c.c.a.t.Request]
> > > > (AgentManager-Handler-1:null) (logid:) Seq 1-5410793477308947404:
> > > > Processing: { Ans: , MgmtId: 90520740759984, via: 1, Ver: v1, Flags:
> > 10,
> > > >
> > >
> >
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"BACKUP","result":"true","details":"Status:
> > > > BACKUP
> > > > ","wait":"0","bypassHostMaintenance":"false"}}] }
> > > > 2022-08-29 17:02:00,750 DEBUG [c.c.a.t.Request]
> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
> > > > 1-5410793477308947404: Received: { Ans: , MgmtId: 90520740759984,
> via:
> > > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer
> } }
> > > > 2022-08-29 17:02:00,751 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f)
> Details
> > > > from executing class com.cloud.agent.api.CheckRouterCommand: Status:
> > > BACKUP
> > > >
> > > > 2022-08-29 17:02:01,362 DEBUG [c.c.s.StatsCollector]
> > > > (Cluster-Worker-839:ctx-e097a8ab) (logid:3deacf65) StatusUpdate from
> > > > 90520740759984, json:
> > > >
> > >
> >
> {"managementServerHostId":1,"managementServerHostUuid":"5615ca5f-0716-4da1-93ec-31f79fd19bbe","collectionTime":"Aug
> > > > 29, 2022, 5:02:01
> > > >
> > >
> >
> PM","sessions":1,"cpuUtilization":0.0,"totalJvmMemoryBytes":324534272,"freeJvmMemoryBytes":68595424,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":103760388,"jvmStartTime":1661681560910,"availableProcessors":6,"loadAverage":0.02,"totalInit":139788288,"totalUsed":474791104,"totalCommitted":557662208,"pid":1522332,"jvmName":"1522332@vm-1-cs-mgmt.aaaaaaaa.bbb
> > >
> >
> ","jvmVendor":"Ubuntu","jvmVersion":"11.0.16+8-post-Ubuntu-0ubuntu120.04","osDistribution":"Ubuntu
> > > > 20.04.5
> > > >
> > >
> >
> LTS","agentCount":5,"heapMemoryUsed":256775312,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":13,"threadsRunnableCount":15,"threadsTerminatedCount":0,"threadsTotalCount":420,"threadsWaitingCount":331,"systemMemoryTotal":8343236608,"systemMemoryFree":2623193088,"systemMemoryUsed":814808,"systemMemoryVirtualSize":7494184960,"logInfo":"/var/log/cloudstack/management/management-server.log
> > > > using: 74M\non disk /dev/vda2 mounted on / (14%
> > > >
> > >
> >
> full)","systemTotalCpuCycles":15870.180000000002,"systemLoadAverages":[0.02,0.03,0.0],"systemCyclesUsage":[4116569,2615101,2716460424],"dbLocal":false,"usageLocal":false,"systemBootTime":"Jul
> > > > 8, 2022, 2:35:55 AM","kernelVersion":"5.4.0-121-generic"}
> > > > 2022-08-29 17:02:01,723 DEBUG [o.a.c.h.HAManagerImpl]
> > > > (BackgroundTaskPollManager-4:ctx-7abb1b5d) (logid:2320950d) HA health
> > > check
> > > > task is running...
> > > > 2022-08-29 17:02:05,417 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > > (Vm-Operations-Cleanup-1:ctx-e741e208) (logid:3f5129e0) VM Operation
> > > Thread
> > > > Running
> > > > 2022-08-29 17:02:05,725 DEBUG [o.a.c.h.HAManagerImpl]
> > > > (BackgroundTaskPollManager-4:ctx-8e7f6766) (logid:710cb3b2) HA health
> > > check
> > > > task is running...
> > > > 2022-08-29 17:02:08,626 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > (AgentManager-Handler-11:null) (logid:) Ping from Routing host
> > > > 6(srv-3-comp.aaaaaaaa.bbb)
> > > >
> > > > 2022-08-29 17:02:09,727 DEBUG [o.a.c.h.HAManagerImpl]
> > > > (BackgroundTaskPollManager-4:ctx-26a5ff07) (logid:09b3c89a) HA health
> > > check
> > > > task is running...
> > > > 2022-08-29 17:02:13,729 DEBUG [o.a.c.h.HAManagerImpl]
> > > > (BackgroundTaskPollManager-4:ctx-b4d4e374) (logid:9048c7a7) HA health
> > > check
> > > > task is running...
> > > > 2022-08-29 17:02:17,731 DEBUG [o.a.c.h.HAManagerImpl]
> > > > (BackgroundTaskPollManager-4:ctx-3db3d208) (logid:7bb73f7b) HA health
> > > check
> > > > task is running...
> > > > 2022-08-29 17:02:21,734 DEBUG [o.a.c.h.HAManagerImpl]
> > > > (BackgroundTaskPollManager-1:ctx-fca1d248) (logid:1ed5381a) HA health
> > > check
> > > > task is running...
> > > > 2022-08-29 17:02:25,737 DEBUG [o.a.c.h.HAManagerImpl]
> > > > (BackgroundTaskPollManager-1:ctx-984e2382) (logid:e6c3d724) HA health
> > > check
> > > > task is running...
> > > > 2022-08-29 17:02:25,759 DEBUG [c.c.s.StatsCollector]
> > > > (secstorage-1:ctx-64c610ed) (logid:6603cd38) Verifying image storage
> > [1].
> > > > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
> > > > 2022-08-29 17:02:25,763 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > > > (secstorage-1:ctx-64c610ed) (logid:6603cd38) Zone [1] is ready to
> > launch
> > > > secondary storage VM.
> > > > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Zone 1 is ready to
> > launch
> > > > console proxy
> > > > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Expand console proxy
> > > standby
> > > > capacity for zone xxxxxxxxxx
> > > > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Allocating console
> proxy
> > > > standby capacity for zone [1].
> > > > 2022-08-29 17:02:26,030 INFO [c.c.c.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Found a stopped
> console
> > > > proxy, starting it. Vm id : 85
> > > > 2022-08-29 17:02:26,031 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) start parameter value
> of
> > > > enterHardwareSetup == <very null> during processing of queued job
> > > > 2022-08-29 17:02:26,037 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Sync job-4289
> execution
> > on
> > > > object VmWorkJobQueue.85
> > > > 2022-08-29 17:02:26,167 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (AsyncJobMgr-Heartbeat-1:ctx-bba6b89d) (logid:f9aed0eb) Execute
> > > sync-queue
> > > > item: SyncQueueItemVO {id:2106, queueId: 2105, contentType: AsyncJob,
> > > > contentId: 4289, lastProcessMsid: 90520740759984, lastprocessNumber:
> 2,
> > > > lastProcessTime: Mon Aug 29 17:02:26 CEST 2022, created: Mon Aug 29
> > > > 17:02:26 CEST 2022}
> > > > 2022-08-29 17:02:26,167 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (AsyncJobMgr-Heartbeat-1:ctx-bba6b89d) (logid:f9aed0eb) Schedule
> queued
> > > > job-4289
> > > > 2022-08-29 17:02:26,172 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
> (logid:f919f9b3)
> > > Add
> > > > job-4289 into job monitoring
> > > > 2022-08-29 17:02:26,181 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
> (logid:ba10fec5)
> > > > Executing AsyncJobVO: {id:4289, userId: 1, accountId: 1,
> instanceType:
> > > > null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
> > > >
> > >
> >
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAVXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
> > > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> > > > result: null, initMsid: 90520740759984, completeMsid: null,
> > lastUpdated:
> > > > null, lastPolled: null, created: Mon Aug 29 17:02:26 CEST 2022,
> > removed:
> > > > null}
> > > > 2022-08-29 17:02:26,181 DEBUG [c.c.v.VmWorkJobDispatcher]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
> (logid:ba10fec5)
> > > Run
> > > > VM work job: com.cloud.vm.VmWorkStart for VM 85, job origin: 3445
> > > > 2022-08-29 17:02:26,183 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > > (logid:ba10fec5) Execute VM work job:
> > > >
> > >
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> > > > 2022-08-29 17:02:26,184 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > > (logid:ba10fec5) orchestrating VM start for 'v-85-VM'
> > > > com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
> > > > 2022-08-29 17:02:26,192 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
> > > > transited from [Stopped] to [Starting] with event [StartRequested].
> > VM's
> > > > original host: null, new host: null, host before state transition:
> null
> > > > 2022-08-29 17:02:26,192 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > > (logid:ba10fec5) Successfully transitioned to start state for VM
> > instance
> > > > {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80",
> > > > type="ConsoleProxy"} reservation id =
> > > 24bf955f-96d8-4f4f-ae33-2cae2f3e0425
> > > > 2022-08-29 17:02:26,198 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > > (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts:
> null
> > > > 2022-08-29 17:02:26,200 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > > (logid:ba10fec5) VM start attempt #1
> > > > 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > > (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
> > > > 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > > (logid:ba10fec5) Trying to allocate a host and storage pools from
> dc:1,
> > > > pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
> > > > 1073741824
> > > > 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > > (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
> > > > 2022-08-29 17:02:26,211 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > > (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
> > > > 2022-08-29 17:02:26,212 DEBUG [c.c.d.FirstFitPlanner]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > > (logid:ba10fec5) Searching all possible resources under this Zone: 1
> > > > 2022-08-29 17:02:26,213 DEBUG [c.c.d.FirstFitPlanner]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > > (logid:ba10fec5) Listing clusters in order of aggregate capacity,
> that
> > > have
> > > > (at least one host with) enough CPU and RAM capacity under this
> Zone: 1
> > > > 2022-08-29 17:02:26,216 DEBUG [c.c.d.FirstFitPlanner]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > > (logid:ba10fec5) Removing from the clusterId list these clusters from
> > > avoid
> > > > set: []
> > > > 2022-08-29 17:02:26,221 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > > (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
> > > > 2022-08-29 17:02:26,222 INFO [c.c.a.m.a.i.FirstFitAllocator]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested
> with
> > > > Custom[UEFI] Boot Type false
> > > > 2022-08-29 17:02:26,223 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in dc: 1
> > > > pod:1 cluster:1
> > > > 2022-08-29 17:02:26,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4
> > hosts
> > > to
> > > > check for allocation: [Host {"id": "7", "name":
> > > "srv-5-comp.aaaaaaaa.bbb",
> > > > "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"},
> Host
> > > > {"id": "1", "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> > > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}, Host
> {"id":
> > > "6",
> > > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> > > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
> {"id":
> > > "5",
> > > > "name": "srv-4-comp.aaaaaaaa.bbb", "uuid":
> > > > "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}]
> > > > 2022-08-29 17:02:26,231 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for
> allocation
> > > > after prioritization: [Host {"id": "7", "name":
> > > "srv-5-comp.aaaaaaaa.bbb",
> > > > "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"},
> Host
> > > > {"id": "1", "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> > > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}, Host
> {"id":
> > > "6",
> > > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> > > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
> {"id":
> > > "5",
> > > > "name": "srv-4-comp.aaaaaaaa.bbb", "uuid":
> > > > "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}]
> > > > 2022-08-29 17:02:26,231 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for speed=500Mhz,
> > > > Ram=1024 MB
> > > > 2022-08-29 17:02:26,231 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
> > > > srv-5-comp.aaaaaaaa.bbb, uuid: 8c753ce1-0253-49c9-9c7f-1f971a719ce6}
> is
> > > KVM
> > > > hypervisor type, no max guest limit check needed
> > > > 2022-08-29 17:02:26,234 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu capability
> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed:
> > 500
> > > > 2022-08-29 17:02:26,234 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has
> > enough
> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> > 1073741824 ,
> > > > cpuOverprovisioningFactor: 1.0
> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> > > 127200
> > > > and CPU after applying overprovisioning: 127200
> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 ,
> Requested
> > > CPU:
> > > > 500
> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
> > > > 189373743104 , Requested RAM: (1.00 GB) 1073741824
> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
> RAM
> > > > available
> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> > > host:
> > > > 7, used: 90100, reserved: 0, actual total: 127200, total with
> > > > overprovisioning: 127200; requested
> cpu:500,alloc_from_last_host?:false
> > > > ,considerReservedCapacity?: true
> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> > > host:
> > > > 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total:
> (250.27
> > > GB)
> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
> > alloc_from_last_host?:
> > > > false , considerReservedCapacity?: true
> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> > adding
> > > to
> > > > list: 7
> > > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
> > > > srv-2-comp.aaaaaaaa.bbb, uuid: 883f7f62-55f2-42f4-8242-16b6847c4204}
> is
> > > KVM
> > > > hypervisor type, no max guest limit check needed
> > > > 2022-08-29 17:02:26,239 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu capability
> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed:
> > 500
> > > > 2022-08-29 17:02:26,239 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has
> > enough
> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> > 1073741824 ,
> > > > cpuOverprovisioningFactor: 1.0
> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> > > 127200
> > > > and CPU after applying overprovisioning: 127200
> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 ,
> Requested
> > > CPU:
> > > > 500
> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
> > > > 229802639360 , Requested RAM: (1.00 GB) 1073741824
> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
> RAM
> > > > available
> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> > > host:
> > > > 1, used: 48200, reserved: 0, actual total: 127200, total with
> > > > overprovisioning: 127200; requested
> cpu:500,alloc_from_last_host?:false
> > > > ,considerReservedCapacity?: true
> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> > > host:
> > > > 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total:
> (250.27
> > > GB)
> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
> > alloc_from_last_host?:
> > > > false , considerReservedCapacity?: true
> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> > adding
> > > to
> > > > list: 1
> > > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
> > > > srv-3-comp.aaaaaaaa.bbb, uuid: 6858713b-7e17-46d8-854d-24a13f9c7001}
> is
> > > KVM
> > > > hypervisor type, no max guest limit check needed
> > > > 2022-08-29 17:02:26,243 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu capability
> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed:
> > 500
> > > > 2022-08-29 17:02:26,243 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has
> > enough
> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> > 1073741824 ,
> > > > cpuOverprovisioningFactor: 1.0
> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> > > 127200
> > > > and CPU after applying overprovisioning: 127200
> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 ,
> Requested
> > > CPU:
> > > > 500
> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
> > > > 204032835584 , Requested RAM: (1.00 GB) 1073741824
> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
> RAM
> > > > available
> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> > > host:
> > > > 6, used: 42900, reserved: 0, actual total: 127200, total with
> > > > overprovisioning: 127200; requested
> cpu:500,alloc_from_last_host?:false
> > > > ,considerReservedCapacity?: true
> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> > > host:
> > > > 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total:
> (250.27
> > > GB)
> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
> > alloc_from_last_host?:
> > > > false , considerReservedCapacity?: true
> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> > adding
> > > to
> > > > list: 6
> > > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
> > > > srv-4-comp.aaaaaaaa.bbb, uuid: 134c625d-15ab-461a-a38d-36fb0b979b5d}
> is
> > > KVM
> > > > hypervisor type, no max guest limit check needed
> > > > 2022-08-29 17:02:26,247 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu capability
> > > > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed:
> > 500
> > > > 2022-08-29 17:02:26,247 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has
> > enough
> > > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> > 1073741824 ,
> > > > cpuOverprovisioningFactor: 1.0
> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> > > 127200
> > > > and CPU after applying overprovisioning: 127200
> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 ,
> Requested
> > > > CPU: 500
> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
> > > > 251009040384 , Requested RAM: (1.00 GB) 1073741824
> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
> RAM
> > > > available
> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> > > host:
> > > > 5, used: 11100, reserved: 0, actual total: 127200, total with
> > > > overprovisioning: 127200; requested
> cpu:500,alloc_from_last_host?:false
> > > > ,considerReservedCapacity?: true
> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> > > host:
> > > > 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total:
> (250.27
> > > GB)
> > > > 268725780480; requested mem: (1.00 GB) 1073741824,
> > alloc_from_last_host?:
> > > > false , considerReservedCapacity?: true
> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> > adding
> > > to
> > > > list: 5
> > > > 2022-08-29 17:02:26,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > > FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator returning 4
> > > > suitable hosts
> > > > 2022-08-29 17:02:26,260 DEBUG [c.c.c.CapacityManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
> > > > transited from [Starting] to [Stopped] with event [OperationFailed].
> > VM's
> > > > original host: null, new host: null, host before state transition:
> null
> > > > 2022-08-29 17:02:26,260 WARN [c.c.v.VirtualMachineManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > > (logid:ba10fec5) Unable to orchestrate start VM instance {id: "85",
> > name:
> > > > "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
> > > > type="ConsoleProxy"} due to [Unable to create deployment, no usable
> > > volumes
> > > > found for the VM: 85].
> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to create
> > > > deployment, no usable volumes found for the VM: 85
> > > > at
> > > >
> > >
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
> > > > at
> > > >
> > >
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
> > > > at
> > > >
> > >
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
> > > > at
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
> > > > at
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> > Source)
> > > > at
> > > >
> > >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > > > at
> > > >
> > >
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > > > at
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > > > at
> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > > > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > > at java.base/java.lang.Thread.run(Thread.java:829)
> > > > 2022-08-29 17:02:26,261 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > > (logid:ba10fec5) Done executing VM work job:
> > > >
> > >
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> > > > 2022-08-29 17:02:26,262 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > > (logid:ba10fec5) Complete async job-4289, jobStatus: FAILED,
> > resultCode:
> > > 0,
> > > > result:
> > > >
> > >
> >
> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODUiLCBuYW1lOiAidi04NS1WTSIsIHV1aWQ6ICIxMTBkODYyMC1jMDI5LTRjYTktOTA4NC0yMTAwOTk3MjVhODAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODVdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
> > > > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > > (logid:ba10fec5) Publish async job-4289 complete on message bus
> > > > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > > (logid:ba10fec5) Wake up jobs related to job-4289
> > > > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > > (logid:ba10fec5) Update db status for job-4289
> > > > 2022-08-29 17:02:26,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > > (logid:ba10fec5) Wake up jobs joined with job-4289 and disjoin all
> > > subjobs
> > > > created from job- 4289
> > > > 2022-08-29 17:02:26,269 DEBUG [c.c.v.VmWorkJobDispatcher]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
> (logid:ba10fec5)
> > > > Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 85, job
> > > > origin: 3445
> > > > 2022-08-29 17:02:26,270 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
> (logid:ba10fec5)
> > > > Done executing com.cloud.vm.VmWorkStart for job-4289
> > > > 2022-08-29 17:02:26,271 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289)
> (logid:ba10fec5)
> > > > Remove job-4289 from job monitoring
> > > > 2022-08-29 17:02:26,277 WARN [c.c.c.ConsoleProxyManagerImpl]
> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Unable to allocate
> > console
> > > > proxy standby capacity for zone [1] due to [Unable to orchestrate
> start
> > > VM
> > > > instance {id: "85", name: "v-85-VM", uuid:
> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> > > [Unable
> > > > to create deployment, no usable volumes found for the VM: 85].].
> > > > com.cloud.utils.exception.CloudRuntimeException: Unable to
> orchestrate
> > > > start VM instance {id: "85", name: "v-85-VM", uuid:
> > > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> > > [Unable
> > > > to create deployment, no usable volumes found for the VM: 85].
> > > > at
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> > > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> > Source)
> > > > at
> > > >
> > >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > > > at
> > > >
> > >
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > > > at
> > > >
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > > > at
> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > > > at
> > > >
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > > > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > > at
> > > >
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > > at java.base/java.lang.Thread.run(Thread.java:829)
> > > > 2022-08-29 17:02:26,277 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) received console proxy
> > > alert
> > > >
> > > >
> > > >
> > > > Am Mo., 29. Aug. 2022 um 16:30 Uhr schrieb Slavka Peleva <
> > > > slavkap@storpool.com>:
> > > >
> > > >> Thanks, Chris, for the information!
> > > >>
> > > >> Are you using custom offerings for the system VMs? Can you share
> more
> > > >> info if yes?
> > > >>
> > > >> About the log, it's not the complete log from where the deployment
> > > >> started. The deployment begins a bit earlier than what you shared.
> Can
> > > you
> > > >> search the management log for `Sync job-3539 execution on object
> > > >> VmWorkJobQueue` to get the `logid` for the volume creation (coloured
> > in
> > > >> blue)? And if possible, share the logs with the found log ID
> > > >>
> > > >> Here is an example from my logs:
> > > >>
> > > >> 2022-08-29 17:07:49,114 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > >> (consoleproxy-1:ctx-a952a3dc) (logid:89157bb9) Sync job-60484
> > execution
> > > >> on object VmWorkJobQueue.4904
> > > >>
> > > >> 2022-08-29 17:07:50,816 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> > > >> (Work-Job-Executor-5:ctx-56172581 job-60479/job-60484 ctx-a900986d)
> > > >> (logid:95d750ae) Execute VM work job:
> > > >>
> > >
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":4904,"handlerName":"VirtualMachineManagerImpl"}
> > > >> 2022-08-29 17:07:50,817 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > >> (Work-Job-Executor-5:ctx-56172581 job-60479/job-60484 ctx-a900986d)
> > > >> (logid:95d750ae) orchestrating VM start for 'v-4904-VM'
> > > >> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
> > > >>
> > > >>
> > > >> On Mon, Aug 29, 2022 at 2:29 PM Vash_X@gmx.de <Va...@gmx.de>
> wrote:
> > > >>
> > > >>> Hi Slavka,
> > > >>>
> > > >>> thats what i did after your suggestion / informatin and where the
> > error
> > > >>> occured.
> > > >>> Deleted / Destroyed the consoleproxy and waited till recreation.
> > Which
> > > >>> didn't worked throwing the error i pasted above.
> > > >>> Sorry for beeing so unspecific.
> > > >>>
> > > >>> Regards,
> > > >>> Chris
> > > >>>
> > > >>> Am Mo., 29. Aug. 2022 um 12:46 Uhr schrieb Slavka Peleva <
> > > >>> slavkap@storpool.com>:
> > > >>>
> > > >>>> Hi Chris,
> > > >>>>
> > > >>>> I mean to recreate the system VMs by destroying them.
> > > >>>>
> > > >>>> On Mon, Aug 29, 2022 at 12:38 PM Vash_X@gmx.de <Va...@gmx.de>
> > wrote:
> > > >>>>
> > > >>>>> Hi Slavka,
> > > >>>>>
> > > >>>>> didn't tried to restart the SystemVMs. Reading through the docs i
> > had
> > > >>>>> the impression that "live-patching" would be enough for this....
> > > >>>>> Anyway - I tried my luck with the console proxy.
> > > >>>>>
> > > >>>>> Which failed utterly. Not starting as it isn't finding any
> useable
> > > >>>>> storage pool. - logs at the end of the mail.
> > > >>>>>
> > > >>>>> I tried deploying a VM using the same storage pool which worked
> > > >>>>> wonderfully.
> > > >>>>> There i noticed that all my custome storage-offerings are
> unuseable
> > > at
> > > >>>>> the moment. They are not accepting any custome storage sizes
> > > anymore...
> > > >>>>> Throwing failures "VM Creation failed. Volume size: -1GB is out
> of
> > > allowed
> > > >>>>> tange. Max:3072 Min:1"...
> > > >>>>> So this is also currently broken....
> > > >>>>>
> > > >>>>> 2022-08-29 10:51:56,189 DEBUG [c.c.v.VmWorkJobDispatcher]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
> > > (logid:ba10fec5) Run
> > > >>>>> VM work job: com.cloud.vm.VmWorkStart for VM 83, job origin: 3445
> > > >>>>> 2022-08-29 10:51:56,191 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> ctx-073f2751)
> > > >>>>> (logid:ba10fec5) Execute VM work job:
> > > >>>>>
> > >
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
> > > >>>>> 2022-08-29 10:51:56,192 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> ctx-073f2751)
> > > >>>>> (logid:ba10fec5) orchestrating VM start for 'v-83-VM'
> > > >>>>> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
> > > >>>>> 2022-08-29 10:51:56,198 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> ctx-073f2751)
> > > >>>>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
> state
> > > >>>>> transited from [Stopped] to [Starting] with event
> [StartRequested].
> > > VM's
> > > >>>>> original host: null, new host: null, host before state
> transition:
> > > null
> > > >>>>> 2022-08-29 10:51:56,198 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> ctx-073f2751)
> > > >>>>> (logid:ba10fec5) Successfully transitioned to start state for VM
> > > instance
> > > >>>>> {id: "83", name: "v-83-VM", uuid:
> > > "b1d1f9c6-eafe-4527-9335-e649c646aab0",
> > > >>>>> type="ConsoleProxy"} reservation id =
> > > 6641ccc0-bcef-4b1a-a1f7-8dd8000626ad
> > > >>>>> 2022-08-29 10:51:56,203 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> ctx-073f2751)
> > > >>>>> (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts:
> > > null
> > > >>>>> 2022-08-29 10:51:56,206 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> ctx-073f2751)
> > > >>>>> (logid:ba10fec5) VM start attempt #1
> > > >>>>> 2022-08-29 10:51:56,207 DEBUG
> [c.c.d.DeploymentPlanningManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> ctx-073f2751)
> > > >>>>> (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
> > > >>>>> 2022-08-29 10:51:56,207 DEBUG
> [c.c.d.DeploymentPlanningManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> ctx-073f2751)
> > > >>>>> (logid:ba10fec5) Trying to allocate a host and storage pools from
> > > dc:1,
> > > >>>>> pod:null,cluster:null, requested cpu: 500, requested ram: (1.00
> GB)
> > > >>>>> 1073741824
> > > >>>>> 2022-08-29 10:51:56,207 DEBUG
> [c.c.d.DeploymentPlanningManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> ctx-073f2751)
> > > >>>>> (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?:
> No
> > > >>>>> 2022-08-29 10:51:56,214 DEBUG
> [c.c.d.DeploymentPlanningManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> ctx-073f2751)
> > > >>>>> (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts:
> null
> > > >>>>> 2022-08-29 10:51:56,215 DEBUG [c.c.d.FirstFitPlanner]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> ctx-073f2751)
> > > >>>>> (logid:ba10fec5) Searching all possible resources under this
> Zone:
> > 1
> > > >>>>> 2022-08-29 10:51:56,216 DEBUG [c.c.d.FirstFitPlanner]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> ctx-073f2751)
> > > >>>>> (logid:ba10fec5) Listing clusters in order of aggregate capacity,
> > > that have
> > > >>>>> (at least one host with) enough CPU and RAM capacity under this
> > > Zone: 1
> > > >>>>> 2022-08-29 10:51:56,219 DEBUG [c.c.d.FirstFitPlanner]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> ctx-073f2751)
> > > >>>>> (logid:ba10fec5) Removing from the clusterId list these clusters
> > > from avoid
> > > >>>>> set: []
> > > >>>>> 2022-08-29 10:51:56,224 DEBUG
> [c.c.d.DeploymentPlanningManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> ctx-073f2751)
> > > >>>>> (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
> > > >>>>> 2022-08-29 10:51:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested
> > > with
> > > >>>>> Custom[UEFI] Boot Type false
> > > >>>>> 2022-08-29 10:51:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in
> > dc: 1
> > > >>>>> pod:1 cluster:1
> > > >>>>> 2022-08-29 10:51:56,227 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has
> 4
> > > hosts to
> > > >>>>> check for allocation: [Host {"id": "7", "name":
> > > "srv-5-comp.blabla.blub",
> > > >>>>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6",
> "type"="Routing"},
> > > Host
> > > >>>>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
> > > >>>>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
> > > {"id": "6",
> > > >>>>> "name": "srv-3-comp.blabla.blub", "uuid":
> > > >>>>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
> > > {"id": "1",
> > > >>>>> "name": "srv-2-comp.blabla.blub", "uuid":
> > > >>>>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> > > >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for
> > > allocation
> > > >>>>> after prioritization: [Host {"id": "7", "name":
> > > "srv-5-comp.blabla.blub",
> > > >>>>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6",
> "type"="Routing"},
> > > Host
> > > >>>>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
> > > >>>>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
> > > {"id": "6",
> > > >>>>> "name": "srv-3-comp.blabla.blub", "uuid":
> > > >>>>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
> > > {"id": "1",
> > > >>>>> "name": "srv-2-comp.blabla.blub", "uuid":
> > > >>>>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> > > >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for
> > speed=500Mhz,
> > > >>>>> Ram=1024 MB
> > > >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
> > > >>>>> srv-5-comp.blabla.blub, uuid:
> 8c753ce1-0253-49c9-9c7f-1f971a719ce6}
> > > is KVM
> > > >>>>> hypervisor type, no max guest limit check needed
> > > >>>>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu
> > capability
> > > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
> > > speed: 500
> > > >>>>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7
> has
> > > enough
> > > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> > > 1073741824 ,
> > > >>>>> cpuOverprovisioningFactor: 1.0
> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
> > CPU:
> > > 127200
> > > >>>>> and CPU after applying overprovisioning: 127200
> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 ,
> > > Requested CPU:
> > > >>>>> 500
> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
> > > >>>>> 189373743104 , Requested RAM: (1.00 GB) 1073741824
> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
> and
> > > RAM
> > > >>>>> available
> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
> > from
> > > host:
> > > >>>>> 7, used: 90100, reserved: 0, actual total: 127200, total with
> > > >>>>> overprovisioning: 127200; requested
> > > cpu:500,alloc_from_last_host?:false
> > > >>>>> ,considerReservedCapacity?: true
> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
> > from
> > > host:
> > > >>>>> 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total:
> > > (250.27 GB)
> > > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
> > > alloc_from_last_host?:
> > > >>>>> false , considerReservedCapacity?: true
> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> > > adding to
> > > >>>>> list: 7
> > > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
> > > >>>>> srv-4-comp.blabla.blub, uuid:
> 134c625d-15ab-461a-a38d-36fb0b979b5d}
> > > is KVM
> > > >>>>> hypervisor type, no max guest limit check needed
> > > >>>>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu
> > capability
> > > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
> > > speed: 500
> > > >>>>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5
> has
> > > enough
> > > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> > > 1073741824 ,
> > > >>>>> cpuOverprovisioningFactor: 1.0
> > > >>>>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
> > CPU:
> > > 127200
> > > >>>>> and CPU after applying overprovisioning: 127200
> > > >>>>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 ,
> > > Requested
> > > >>>>> CPU: 500
> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
> > > >>>>> 251009040384 , Requested RAM: (1.00 GB) 1073741824
> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
> and
> > > RAM
> > > >>>>> available
> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
> > from
> > > host:
> > > >>>>> 5, used: 11100, reserved: 0, actual total: 127200, total with
> > > >>>>> overprovisioning: 127200; requested
> > > cpu:500,alloc_from_last_host?:false
> > > >>>>> ,considerReservedCapacity?: true
> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
> > from
> > > host:
> > > >>>>> 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total:
> > > (250.27 GB)
> > > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
> > > alloc_from_last_host?:
> > > >>>>> false , considerReservedCapacity?: true
> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> > > adding to
> > > >>>>> list: 5
> > > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
> > > >>>>> srv-3-comp.blabla.blub, uuid:
> 6858713b-7e17-46d8-854d-24a13f9c7001}
> > > is KVM
> > > >>>>> hypervisor type, no max guest limit check needed
> > > >>>>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu
> > capability
> > > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
> > > speed: 500
> > > >>>>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6
> has
> > > enough
> > > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> > > 1073741824 ,
> > > >>>>> cpuOverprovisioningFactor: 1.0
> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
> > CPU:
> > > 127200
> > > >>>>> and CPU after applying overprovisioning: 127200
> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 ,
> > > Requested CPU:
> > > >>>>> 500
> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
> > > >>>>> 204032835584 , Requested RAM: (1.00 GB) 1073741824
> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
> and
> > > RAM
> > > >>>>> available
> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
> > from
> > > host:
> > > >>>>> 6, used: 42900, reserved: 0, actual total: 127200, total with
> > > >>>>> overprovisioning: 127200; requested
> > > cpu:500,alloc_from_last_host?:false
> > > >>>>> ,considerReservedCapacity?: true
> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
> > from
> > > host:
> > > >>>>> 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total:
> > > (250.27 GB)
> > > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
> > > alloc_from_last_host?:
> > > >>>>> false , considerReservedCapacity?: true
> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> > > adding to
> > > >>>>> list: 6
> > > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
> > > >>>>> srv-2-comp.blabla.blub, uuid:
> 883f7f62-55f2-42f4-8242-16b6847c4204}
> > > is KVM
> > > >>>>> hypervisor type, no max guest limit check needed
> > > >>>>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu
> > capability
> > > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
> > > speed: 500
> > > >>>>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1
> has
> > > enough
> > > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> > > 1073741824 ,
> > > >>>>> cpuOverprovisioningFactor: 1.0
> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
> > CPU:
> > > 127200
> > > >>>>> and CPU after applying overprovisioning: 127200
> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 ,
> > > Requested CPU:
> > > >>>>> 500
> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
> > > >>>>> 229802639360 , Requested RAM: (1.00 GB) 1073741824
> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU
> and
> > > RAM
> > > >>>>> available
> > > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
> > from
> > > host:
> > > >>>>> 1, used: 48200, reserved: 0, actual total: 127200, total with
> > > >>>>> overprovisioning: 127200; requested
> > > cpu:500,alloc_from_last_host?:false
> > > >>>>> ,considerReservedCapacity?: true
> > > >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
> > from
> > > host:
> > > >>>>> 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total:
> > > (250.27 GB)
> > > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
> > > alloc_from_last_host?:
> > > >>>>> false , considerReservedCapacity?: true
> > > >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> > > adding to
> > > >>>>> list: 1
> > > >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator
> > returning 4
> > > >>>>> suitable hosts
> > > >>>>> 2022-08-29 10:51:56,260 DEBUG [c.c.c.CapacityManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> ctx-073f2751)
> > > >>>>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
> state
> > > >>>>> transited from [Starting] to [Stopped] with event
> > [OperationFailed].
> > > VM's
> > > >>>>> original host: null, new host: null, host before state
> transition:
> > > null
> > > >>>>> 2022-08-29 10:51:56,260 WARN [c.c.v.VirtualMachineManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> ctx-073f2751)
> > > >>>>> (logid:ba10fec5) Unable to orchestrate start VM instance {id:
> "83",
> > > name:
> > > >>>>> "v-83-VM", uuid: "b1d1f9c6-eafe-4527-9335-e649c646aab0",
> > > >>>>> type="ConsoleProxy"} due to [Unable to create deployment, no
> usable
> > > volumes
> > > >>>>> found for the VM: 83].
> > > >>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to create
> > > >>>>> deployment, no usable volumes found for the VM: 83
> > > >>>>> at
> > > >>>>>
> > >
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
> > > >>>>> at
> > > >>>>>
> > >
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
> > > >>>>> at
> > > >>>>>
> > >
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
> > > >>>>> at
> > > >>>>>
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
> > > >>>>> at
> > > >>>>>
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
> > > >>>>> at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> > > >>>>> Source)
> > > >>>>> at
> > > >>>>>
> > >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > >>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > > >>>>> at
> > > >>>>>
> > >
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > > >>>>> at
> > > >>>>>
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > > >>>>> at
> > > >>>>>
> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > > >>>>> at
> > > >>>>>
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > > >>>>> at
> > > >>>>>
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > > >>>>> at
> > > >>>>>
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > > >>>>> at
> > > >>>>>
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > > >>>>> at
> > > >>>>>
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > > >>>>> at
> > > >>>>>
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > > >>>>> at
> > > >>>>>
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > > >>>>> at
> > > >>>>>
> > >
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > > >>>>> at
> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > > >>>>> at
> > > >>>>>
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > >>>>> at
> > > >>>>>
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > >>>>> at java.base/java.lang.Thread.run(Thread.java:829)
> > > >>>>> 2022-08-29 10:51:56,262 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> ctx-073f2751)
> > > >>>>> (logid:ba10fec5) Done executing VM work job:
> > > >>>>>
> > >
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
> > > >>>>> 2022-08-29 10:51:56,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> ctx-073f2751)
> > > >>>>> (logid:ba10fec5) Complete async job-3539, jobStatus: FAILED,
> > > resultCode: 0,
> > > >>>>> result:
> > > >>>>>
> > >
> >
> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODMiLCBuYW1lOiAidi04My1WTSIsIHV1aWQ6ICJiMWQxZjljNi1lYWZlLTQ1MjctOTMzNS1lNjQ5YzY0NmFhYjAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODNdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
> > > >>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> ctx-073f2751)
> > > >>>>> (logid:ba10fec5) Publish async job-3539 complete on message bus
> > > >>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> ctx-073f2751)
> > > >>>>> (logid:ba10fec5) Wake up jobs related to job-3539
> > > >>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> ctx-073f2751)
> > > >>>>> (logid:ba10fec5) Update db status for job-3539
> > > >>>>> 2022-08-29 10:51:56,266 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539
> ctx-073f2751)
> > > >>>>> (logid:ba10fec5) Wake up jobs joined with job-3539 and disjoin
> all
> > > subjobs
> > > >>>>> created from job- 3539
> > > >>>>> 2022-08-29 10:51:56,276 DEBUG [c.c.v.VmWorkJobDispatcher]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
> > > (logid:ba10fec5) Done
> > > >>>>> with run of VM work job: com.cloud.vm.VmWorkStart for VM 83, job
> > > origin:
> > > >>>>> 3445
> > > >>>>> 2022-08-29 10:51:56,276 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
> > > (logid:ba10fec5) Done
> > > >>>>> executing com.cloud.vm.VmWorkStart for job-3539
> > > >>>>> 2022-08-29 10:51:56,278 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
> > > (logid:ba10fec5)
> > > >>>>> Remove job-3539 from job monitoring
> > > >>>>> 2022-08-29 10:51:56,285 WARN [c.c.c.ConsoleProxyManagerImpl]
> > > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Unable to allocate
> > > console
> > > >>>>> proxy standby capacity for zone [1] due to [Unable to orchestrate
> > > start VM
> > > >>>>> instance {id: "83", name: "v-83-VM", uuid:
> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due
> to
> > > [Unable
> > > >>>>> to create deployment, no usable volumes found for the VM: 83].].
> > > >>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to
> > > orchestrate
> > > >>>>> start VM instance {id: "83", name: "v-83-VM", uuid:
> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due
> to
> > > [Unable
> > > >>>>> to create deployment, no usable volumes found for the VM: 83].
> > > >>>>> at
> > > >>>>>
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> > > >>>>> at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> > > >>>>> Source)
> > > >>>>> at
> > > >>>>>
> > >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > >>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > > >>>>> at
> > > >>>>>
> > >
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > > >>>>> at
> > > >>>>>
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > > >>>>> at
> > > >>>>>
> > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > > >>>>> at
> > > >>>>>
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > > >>>>> at
> > > >>>>>
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > > >>>>> at
> > > >>>>>
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > > >>>>> at
> > > >>>>>
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > > >>>>> at
> > > >>>>>
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > > >>>>> at
> > > >>>>>
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > > >>>>> at
> > > >>>>>
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > > >>>>> at
> > > >>>>>
> > >
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > > >>>>> at
> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > > >>>>> at
> > > >>>>>
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > >>>>> at
> > > >>>>>
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > >>>>> at java.base/java.lang.Thread.run(Thread.java:829)
> > > >>>>> 2022-08-29 10:51:56,287 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) received console
> > > proxy alert
> > > >>>>> 2022-08-29 10:51:56,288 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Console proxy
> > creation
> > > >>>>> failure, zone: blabla
> > > >>>>> 2022-08-29 10:51:56,289 WARN [c.c.a.AlertManagerImpl]
> > > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) alertType=[10]
> > > >>>>> dataCenterId=[1] podId=[null] clusterId=[null] message=[Console
> > proxy
> > > >>>>> creation failure. zone: blabla, error details: Unable to
> > orchestrate
> > > start
> > > >>>>> VM instance {id: "83", name: "v-83-VM", uuid:
> > > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due
> to
> > > [Unable
> > > >>>>> to create deployment, no usable volumes found for the VM: 83].].
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> Am Mo., 29. Aug. 2022 um 09:40 Uhr schrieb Slavka Peleva
> > > >>>>> <sl...@storpool.com.invalid>:
> > > >>>>>
> > > >>>>>> Hi Chris,
> > > >>>>>>
> > > >>>>>> Did you recreate the system VMs? In the 4.17 version, the
> > > >>>>>> `systemvm.iso` is
> > > >>>>>> deprecated. In its place, you should have the file on agents
> > > >>>>>> `/usr/share/cloudstack-common/vms/agent.zip`.
> > > >>>>>> Can you share the complete log if the system VMs are newly
> created
> > > >>>>>> after
> > > >>>>>> the upgrade?
> > > >>>>>>
> > > >>>>>> Best regards,
> > > >>>>>> Slavka
> > > >>>>>>
> > > >>>>>> On Sun, Aug 28, 2022 at 2:16 PM Vash_X@gmx.de <Va...@gmx.de>
> > > wrote:
> > > >>>>>>
> > > >>>>>> > Hi everyone,
> > > >>>>>> >
> > > >>>>>> > faceing some challanges again after upgradeing to 4.17.
> > > >>>>>> > Did as explained in the docs.
> > > >>>>>> >
> > > >>>>>> > Afterwards I am now not able to migrate system VMs to
> different
> > > >>>>>> hosts.
> > > >>>>>> > Getting the error
> > > >>>>>> > "Exception during migrate: org.libvirt.LibvirtException:
> Cannot
> > > >>>>>> access
> > > >>>>>> > storage file '/usr/share/cloudstack-common/vms/systemvm.iso':
> No
> > > >>>>>> such file
> > > >>>>>> > or directory"
> > > >>>>>> >
> > > >>>>>> > Checked the exitstence of the file on the servers:
> > > >>>>>> > Mangement-Server has a "systemvm.iso.bak"
> > > >>>>>> > Hosts - no "systemvm.iso" at all present.
> > > >>>>>> >
> > > >>>>>> > Maybe someone has an idea on how to move on with this?
> > > >>>>>> >
> > > >>>>>> > Regards,
> > > >>>>>> > Chris
> > > >>>>>> >
> > > >>>>>>
> > > >>>>>
> > >
> >
>
Re: Error migrate System VMs after upgrageing to 4.17
Posted by "Vash_X@gmx.de" <Va...@gmx.de>.
Hi Nicolas,
> What is the value for the setting
‘system.vm.public.ip.reservation.mode.strictness’?
This is set to 'true'.
> Can you please check if this query gets any result?
Yes the query gives me results - all availeable "public" ip addresses of
the reserved range for the systemVMs
# public_ip_address vlan_db_id vlan_id
1 10.129.0.10 9 1000
2 10.129.0.11 9 1000
3 10.129.0.12 9 1000
4 10.129.0.13 9 1000
5 10.129.0.14 9 1000
6 10.129.0.15 9 1000
7 10.129.0.16 9 1000
8 10.129.0.18 9 1000
9 10.129.0.19 9 1000
Regarding the vlanId id=[3, 10, 13, 14]
I looked up my table "vlan" and am quite shocked:
the ids 3, 13, 14 have a value for "range removed", so i don't know why the
console proxy is looking for these networks.
the id "10" is is still active - however the ipv4 range is not "the right
one". That is assigned as the public ip-range for the root-account but not
for use of systemvms...
Am Di., 30. Aug. 2022 um 16:43 Uhr schrieb Nicolas Vazquez <
Nicolas.Vazquez@shapeblue.com>:
> Hi Chris,
>
> What is the value for the setting
> ‘system.vm.public.ip.reservation.mode.strictness’?
>
> Looks like the error is not being able to get an IP address for zone id=1,
> vlanId id=[3, 10, 13, 14].
> If I understood correctly you have defined a range for system VMs? Can you
> please check if this query gets any result?
>
> select i.public_ip_address, i.vlan_db_id, v.vlan_id from user_ip_address i
> join vlan v on (i.vlan_db_id = v.id)
> where i.forsystemvms = 1 and i.allocated is null and i.state = 'Free' and
> v.removed is null and i.data_center_id = 1;
>
> Regards,
> Nicolas Vazquez
>
>
> From: Vash_X@gmx.de <Va...@gmx.de>
> Date: Tuesday, 30 August 2022 at 09:38
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>, Slavka
> Peleva <sl...@storpool.com>
> Subject: Re: Error migrate System VMs after upgrageing to 4.17
> Hi Slavka,
>
> that was my first thought, too. I checked beforehand and confirmed with
> your provided SQL Statements:
> *`select * from user_ip_address where forsystemvms = 1;`* <---- returns my
> defined (Public) IPs out of the correct range for SystemVMs
> *`SELECT * FROM clou*d.user_ip_address where allocated IS NULL AND removed
> IS NULL; <-- All IPs shown beside the one which is currently in use for the
> running ssvm. Meaning from my point of view "free" for allocation for other
> system VMs.
>
> Querys performed directly on the DB-Server.
> I can't see a reason why there shouldn't be any IPs left for allocation.
>
> Also it is interesting why the "reasoning" for the failing deployment
> changes from "no IP availeable" to "no useable volumes found".
> How ever i can't see anything missing at the moment, what makes me even
> more puzzeled.
>
> Regards
> Chris
>
> Am Di., 30. Aug. 2022 um 13:31 Uhr schrieb Slavka Peleva
> <sl...@storpool.com.invalid>:
>
> > Hi Chris,
> >
> > The deployment fails because CS cannot find a free IP address to allocate
> > for the system VM. Can you check if you explicitly added a range for the
> > system VMs, or you can check with this `select * from user_ip_address
> where
> > forsystemvms = 1;` If you don't have any ranges, you can get free IPs
> > with `SELECT
> > * FROM cloud.user_ip_address where allocated IS NULL AND removed IS
> NULL;`
> > . To get the exact query executed by CS, you need to increase the log
> level
> > to TRACE.
> >
> > Best regards,
> > Slavka
> >
>
>
>
> > On Mon, Aug 29, 2022 at 8:03 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
> >
> > > Hi Slavka,
> > >
> > > no, i am not using any kind of custome offerings.
> > >
> > > I try to provide a more detailled log this time.
> > > Situation:
> > > v-83-VM was the "broken" consoleproxy. So the following log starts with
> > > it's expunging.
> > > Afterwards the new consoleproxy shall be deployed - v-85-VM.
> > >
> > > This first deployment (logid:dcfe0999) fails due to Insufficent address
> > > capacity - which i can't really understand or see in the database,
> > checking
> > > the availeable IPs for systemVMs (given that i might looking up the
> wrong
> > > tables....).
> > > How ever CS is working as expected trying to deploy the console proxy
> > > (logid:440fd101) Sync job-4288).
> > > That job fails with "Unable to create deployment, no usable volumes
> found
> > > for the VM: 85".
> > > Afterwards i have another try to start the consoleproxy (job-4289)
> which
> > > fails for the same reason. And thats how things go on in the log
> file...
> > >
> > >
> > > 2022-08-29 17:01:16,224 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> (logid:5fe8559a)
> > > Cleaning up hypervisor data structures (ex. SRs in XenServer) for
> managed
> > > storage. Data from VM instance {id: "83", name: "v-83-VM", uuid:
> > > "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}.
> > > 2022-08-29 17:01:16,225 DEBUG [o.a.c.e.o.VolumeOrchestrator]
> > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> (logid:5fe8559a)
> > > Cleaning storage for vm: 83
> > > 2022-08-29 17:01:16,233 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> (logid:5fe8559a)
> > > Expunged VM instance {id: "83", name: "v-83-VM", uuid:
> > > "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
> > > 2022-08-29 17:01:16,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> (logid:5fe8559a)
> > > Complete async job-4286, jobStatus: SUCCEEDED, resultCode: 0, result:
> > >
> >
> org.apache.cloudstack.api.response.SystemVmResponse/systemvm/{"id":"b1d1f9c6-eafe-4527-9335-e649c646aab0","systemvmtype":"consoleproxy","zoneid":"48e2e928-3300-43b5-8e3a-d9b358022502","zonename":"xxxxxxxxxx","dns1":"10.129.0.4","dns2":"195.10.195.195","name":"v-83-VM","templateid":"c20799e4-a990-449a-aae6-4b67374c7aff","templatename":"systemvm-kvm-4.17.0","created":"2022-08-29T10:38:26+0200","state":"Stopped","isdynamicallyscalable":"false","hasannotations":"false"}
> > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> (logid:5fe8559a)
> > > Publish async job-4286 complete on message bus
> > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> (logid:5fe8559a)
> > > Wake up jobs related to job-4286
> > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> (logid:5fe8559a)
> > > Update db status for job-4286
> > > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2)
> (logid:5fe8559a)
> > > Wake up jobs joined with job-4286 and disjoin all subjobs created from
> > job-
> > > 4286
> > > 2022-08-29 17:01:16,260 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (API-Job-Executor-35:ctx-29e06f28 job-4286) (logid:5fe8559a) Done
> > executing
> > > org.apache.cloudstack.api.command.admin.systemvm.DestroySystemVmCmd for
> > > job-4286
> > > 2022-08-29 17:01:16,260 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > > (API-Job-Executor-35:ctx-29e06f28 job-4286) (logid:5fe8559a) Remove
> > > job-4286 from job monitoring
> > > 2022-08-29 17:01:17,691 DEBUG [o.a.c.h.HAManagerImpl]
> > > (BackgroundTaskPollManager-6:ctx-6af18142) (logid:fe263a70) HA health
> > check
> > > task is running...
> > >
> > > 2022-08-29 17:01:21,694 DEBUG [o.a.c.h.HAManagerImpl]
> > > (BackgroundTaskPollManager-6:ctx-89b1442f) (logid:0efc40b2) HA health
> > check
> > > task is running...
> > > 2022-08-29 17:01:25,698 DEBUG [o.a.c.h.HAManagerImpl]
> > > (BackgroundTaskPollManager-5:ctx-62313d05) (logid:7073663f) HA health
> > check
> > > task is running...
> > > 2022-08-29 17:01:25,767 DEBUG [c.c.s.StatsCollector]
> > > (secstorage-1:ctx-c80e5a6e) (logid:f8e3eb60) Verifying image storage
> [1].
> > > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
> > > 2022-08-29 17:01:25,773 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > > (secstorage-1:ctx-c80e5a6e) (logid:f8e3eb60) Zone [1] is ready to
> launch
> > > secondary storage VM.
> > > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Zone 1 is ready to
> launch
> > > console proxy
> > > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Expand console proxy
> > standby
> > > capacity for zone xxxxxxxxxx
> > > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating console proxy
> > > standby capacity for zone [1].
> > > 2022-08-29 17:01:26,032 INFO [c.c.c.ConsoleProxyManagerImpl]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) No stopped console proxy
> > is
> > > available, need to allocate a new console proxy
> > > 2022-08-29 17:01:26,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Assign console proxy
> from
> > a
> > > newly started instance for request from data center : 1
> > > 2022-08-29 17:01:26,056 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing network
> > > configuration for offering [Network Offering
> > > [1-Public-System-Public-Network]: Network {"id": 200, "name": "null",
> > > "uuid": "1b0bf965-a391-478d-833d-3e37e4a15eb8", "networkofferingid": 1}
> > > 2022-08-29 17:01:26,056 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
> > > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account {"id": 1,
> > > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
> > > 2022-08-29 17:01:26,064 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing network
> > > configuration for offering [Network Offering
> > > [3-Control-System-Control-Network]: Network {"id": 202, "name": "null",
> > > "uuid": "2af42518-fbf4-4a35-9d6b-370e6ad315c7", "networkofferingid": 3}
> > > 2022-08-29 17:01:26,064 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
> > > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account {"id": 1,
> > > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
> > > 2022-08-29 17:01:26,069 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing network
> > > configuration for offering [Network Offering
> > > [2-Management-System-Management-Network]: Network {"id": 201, "name":
> > > "null", "uuid": "53711d87-a3c6-4f33-bfce-c30e0de72760",
> > > "networkofferingid": 2}
> > > 2022-08-29 17:01:26,069 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
> > > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account {"id": 1,
> > > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
> > > 2022-08-29 17:01:26,077 INFO [c.c.v.VirtualMachineManagerImpl]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) allocating virtual
> machine
> > > from template:c20799e4-a990-449a-aae6-4b67374c7aff with
> hostname:v-85-VM
> > > and 3 networks
> > > 2022-08-29 17:01:26,079 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating entries for
> VM:
> > > VM instance {id: "85", name: "v-85-VM", uuid:
> > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
> > > 2022-08-29 17:01:26,082 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating nics for VM
> > > instance {id: "85", name: "v-85-VM", uuid:
> > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
> > > 2022-08-29 17:01:26,083 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating nic for vm VM
> > > instance {id: "85", name: "v-85-VM", uuid:
> > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} in network
> > > Network {"id": 200, "name": "null", "uuid":
> > > "1b0bf965-a391-478d-833d-3e37e4a15eb8", "networkofferingid": 1} with
> > > requested profile NicProfile
> > >
> >
> {"broadcastUri":null,"iPv4Address":null,"id":0,"reservationId":null,"vmId":0}
> > > 2022-08-29 17:01:26,089 WARN [c.c.n.IpAddressManagerImpl]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to get ip address
> > in
> > > zone id=1, vlanId id=[3, 10, 13, 14]
> > > 2022-08-29 17:01:26,090 DEBUG [c.c.u.d.T.Transaction]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Rolling back the
> > > transaction: Time = 8 Name = consoleproxy-1; called by
> > >
> >
> -TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-Transaction.execute:38-IpAddressManagerImpl.listAvailablePublicIps:824-IpAddressManagerImpl.fetchNewPublicIp:811-IpAddressManagerImpl.fetchNewPublicIp:803-IpAddressManagerImpl.assignPublicIpAddress:784-PublicNetworkGuru.getIp:125-PublicNetworkGuru.allocate:172-NetworkOrchestrator.allocateNic:973-NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice:823
> > > 2022-08-29 17:01:26,090 WARN [c.c.c.ConsoleProxyManagerImpl]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to allocate proxy
> > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone
> [1]
> > > due to [Insufficient address capacity].
> > > com.cloud.exception.InsufficientAddressCapacityException: Insufficient
> > > address capacityScope=interface com.cloud.dc.DataCenter; id=1
> > > at
> > >
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
> > > at
> > >
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
> > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> > > at
> > >
> >
> com.cloud.network.IpAddressManagerImpl.listAvailablePublicIps(IpAddressManagerImpl.java:824)
> > > at
> > >
> >
> com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl.java:811)
> > > at
> > >
> >
> com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl.java:803)
> > > at
> > >
> >
> com.cloud.network.IpAddressManagerImpl.assignPublicIpAddress(IpAddressManagerImpl.java:784)
> > > at
> > >
> >
> com.cloud.network.guru.PublicNetworkGuru.getIp(PublicNetworkGuru.java:125)
> > > at
> > >
> >
> com.cloud.network.guru.PublicNetworkGuru.allocate(PublicNetworkGuru.java:172)
> > > at
> > >
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocateNic(NetworkOrchestrator.java:973)
> > > at
> > >
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice(NetworkOrchestrator.java:823)
> > > at
> > >
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$3.doInTransactionWithoutResult(NetworkOrchestrator.java:797)
> > > at
> > >
> >
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
> > > at
> > >
> >
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
> > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> > > at
> > >
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocate(NetworkOrchestrator.java:778)
> > > at
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl$1.doInTransactionWithoutResult(VirtualMachineManagerImpl.java:478)
> > > at
> > >
> >
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
> > > at
> > >
> >
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
> > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> > > at
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:469)
> > > at
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:528)
> > > at
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:700)
> > > at
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
> > > at
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
> > > at
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
> > > at
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
> > > at
> > com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
> > > at
> > >
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
> > > at
> > >
> >
> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > > at
> > >
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > > at
> > >
> >
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> > > at
> > >
> >
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> > > at
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > at
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > at java.base/java.lang.Thread.run(Thread.java:829)
> > > 2022-08-29 17:01:26,091 WARN [c.c.c.ConsoleProxyManagerImpl]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to allocate
> console
> > > proxy standby capacity for zone [1] due to [Unable to allocate proxy
> > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone
> [1]
> > > due to [Insufficient address capacity].].
> > > com.cloud.utils.exception.CloudRuntimeException: Unable to allocate
> proxy
> > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone
> [1]
> > > due to [Insufficient address capacity].
> > > at
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:704)
> > > at
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
> > > at
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
> > > at
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
> > > at
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
> > > at
> > com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
> > > at
> > >
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
> > > at
> > >
> >
> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > > at
> > >
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > > at
> > >
> >
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> > > at
> > >
> >
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> > > at
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > at
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > at java.base/java.lang.Thread.run(Thread.java:829)
> > > Caused by: com.cloud.exception.InsufficientAddressCapacityException:
> > > Insufficient address capacityScope=interface com.cloud.dc.DataCenter;
> > id=1
> > > at
> > >
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
> > > at
> > >
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
> > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> > > ... 38 more
> > > 2022-08-29 17:01:26,091 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) received console proxy
> > alert
> > > 2022-08-29 17:01:26,092 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Console proxy creation
> > > failure, zone: xxxxxxxxxx
> > > 2022-08-29 17:01:26,093 WARN [c.c.a.AlertManagerImpl]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) alertType=[10]
> > > dataCenterId=[1] podId=[null] clusterId=[null] message=[Console proxy
> > > creation failure. zone: xxxxxxxxxx, error details: Unable to allocate
> > proxy
> > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone
> [1]
> > > due to [Insufficient address capacity].].
> > > 2022-08-29 17:01:26,096 WARN [c.c.a.AlertManagerImpl]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) No recipients set in
> > global
> > > setting 'alert.email.addresses', skipping sending alert with subject
> > > [Console proxy creation failure. zone: xxxxxxxxxx, error details:
> Unable
> > to
> > > allocate proxy [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone
> [1]
> > > due to [Insufficient address capacity].] and content [Console proxy
> > > creation failure (zone xxxxxxxxxx)].
> > > 2022-08-29 17:01:26,096 WARN [c.c.v.SystemVmLoadScanner]
> > > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unexpected exception
> > Unable
> > > to allocate proxy [Console VM instance {id: "85", name: "v-85-VM",
> uuid:
> > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone
> [1]
> > > due to [Insufficient address capacity].
> > > com.cloud.utils.exception.CloudRuntimeException: Unable to allocate
> proxy
> > > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone
> [1]
> > > due to [Insufficient address capacity].
> > > at
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:704)
> > > at
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
> > > at
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
> > > at
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
> > > at
> > >
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
> > > at
> > com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
> > > at
> > >
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
> > > at
> > >
> >
> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > > at
> > >
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > > at
> > >
> >
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> > > at
> > >
> >
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> > > at
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > at
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > at java.base/java.lang.Thread.run(Thread.java:829)
> > > Caused by: com.cloud.exception.InsufficientAddressCapacityException:
> > > Insufficient address capacityScope=interface com.cloud.dc.DataCenter;
> > id=1
> > > at
> > >
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
> > > at
> > >
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
> > > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> > > ... 38 more
> > > 2022-08-29 17:01:29,700 DEBUG [o.a.c.h.HAManagerImpl]
> > > (BackgroundTaskPollManager-5:ctx-994465bd) (logid:1e2822fd) HA health
> > check
> > > task is running...
> > > 2022-08-29 17:01:30,444 DEBUG
> > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 2 routers
> to
> > > update status.
> > > 2022-08-29 17:01:30,449 DEBUG
> > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 1 VPC's to
> > > update Redundant State.
> > > 2022-08-29 17:01:30,451 DEBUG
> > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 0 networks
> to
> > > update RvR status.
> > > 2022-08-29 17:01:30,459 DEBUG [c.c.a.t.Request]
> > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
> > > 6-4833769775052102463: Sending { Cmd , MgmtId: 90520740759984, via:
> > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
> > router.name
> >
> ":"r-79-VM","router.ip":"169.254.144.198"},"wait":"30","bypassHostMaintenance":"false"}}]
> > > }
> > > 2022-08-29 17:01:30,643 DEBUG [c.c.a.t.Request]
> > > (AgentManager-Handler-12:null) (logid:) Seq 6-4833769775052102463:
> > > Processing: { Ans: , MgmtId: 90520740759984, via: 6, Ver: v1, Flags:
> 10,
> > >
> >
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"PRIMARY","result":"true","details":"Status:
> > > PRIMARY
> > > ","wait":"0","bypassHostMaintenance":"false"}}] }
> > > 2022-08-29 17:01:30,643 DEBUG [c.c.a.t.Request]
> > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
> > > 6-4833769775052102463: Received: { Ans: , MgmtId: 90520740759984, via:
> > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer } }
> > > 2022-08-29 17:01:30,643 DEBUG [c.c.a.m.AgentManagerImpl]
> > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Details
> > from
> > > executing class com.cloud.agent.api.CheckRouterCommand: Status: PRIMARY
> > >
> > > 2022-08-29 17:01:30,655 DEBUG [c.c.a.t.Request]
> > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
> > > 1-5410793477308947401: Sending { Cmd , MgmtId: 90520740759984, via:
> > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
> > router.name
> >
> ":"r-80-VM","router.ip":"169.254.217.200"},"wait":"30","bypassHostMaintenance":"false"}}]
> > > }
> > > 2022-08-29 17:01:30,796 DEBUG [c.c.a.t.Request]
> > > (AgentManager-Handler-7:null) (logid:) Seq 1-5410793477308947401:
> > > Processing: { Ans: , MgmtId: 90520740759984, via: 1, Ver: v1, Flags:
> 10,
> > >
> >
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"BACKUP","result":"true","details":"Status:
> > > BACKUP
> > > ","wait":"0","bypassHostMaintenance":"false"}}] }
> > > 2022-08-29 17:01:30,796 DEBUG [c.c.a.t.Request]
> > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
> > > 1-5410793477308947401: Received: { Ans: , MgmtId: 90520740759984, via:
> > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer } }
> > > 2022-08-29 17:01:30,796 DEBUG [c.c.a.m.AgentManagerImpl]
> > > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Details
> > from
> > > executing class com.cloud.agent.api.CheckRouterCommand: Status: BACKUP
> > >
> > > 2022-08-29 17:01:33,703 DEBUG [o.a.c.h.HAManagerImpl]
> > > (BackgroundTaskPollManager-5:ctx-70bca202) (logid:b332dbe0) HA health
> > check
> > > task is running...
> > > 2022-08-29 17:01:34,083 DEBUG [c.c.s.StatsCollector]
> > > (StatsCollector-6:ctx-213e2ecc) (logid:aa0584c9) AutoScaling Monitor is
> > > running...
> > > 2022-08-29 17:01:37,707 DEBUG [o.a.c.h.HAManagerImpl]
> > > (BackgroundTaskPollManager-6:ctx-57493517) (logid:5093e75f) HA health
> > check
> > > task is running...
> > > 2022-08-29 17:01:38,640 DEBUG [c.c.s.StatsCollector]
> > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) HostStatsCollector is
> > > running...
> > > 2022-08-29 17:01:38,655 DEBUG [c.c.a.t.Request]
> > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
> > 1-5410793477308947402:
> > > Received: { Ans: , MgmtId: 90520740759984, via:
> > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer }
> }
> > > 2022-08-29 17:01:38,707 DEBUG [c.c.a.t.Request]
> > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
> > 5-8959067033723473395:
> > > Received: { Ans: , MgmtId: 90520740759984, via:
> > > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer }
> }
> > > 2022-08-29 17:01:38,717 DEBUG [c.c.a.t.Request]
> > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
> > 6-4833769775052102464:
> > > Received: { Ans: , MgmtId: 90520740759984, via:
> > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer }
> }
> > > 2022-08-29 17:01:38,725 DEBUG [c.c.a.t.Request]
> > > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
> > 7-295548725546192472:
> > > Received: { Ans: , MgmtId: 90520740759984, via:
> > > 7(srv-5-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer }
> }
> > > 2022-08-29 17:01:41,709 DEBUG [o.a.c.h.HAManagerImpl]
> > > (BackgroundTaskPollManager-5:ctx-a65ae462) (logid:bc2b7320) HA health
> > check
> > > task is running...
> > > 2022-08-29 17:01:45,712 DEBUG [o.a.c.h.HAManagerImpl]
> > > (BackgroundTaskPollManager-2:ctx-05491ace) (logid:b2232c4d) HA health
> > check
> > > task is running...
> > > 2022-08-29 17:01:47,019 DEBUG [c.c.a.m.AgentManagerImpl]
> > > (AgentManager-Handler-14:null) (logid:) Ping from Routing host
> > > 7(srv-5-comp.aaaaaaaa.bbb)
> > >
> > > 2022-08-29 17:01:49,715 DEBUG [o.a.c.h.HAManagerImpl]
> > > (BackgroundTaskPollManager-2:ctx-7943581c) (logid:504c98ff) HA health
> > check
> > > task is running...
> > > 2022-08-29 17:01:51,881 DEBUG [c.c.a.m.AgentManagerImpl]
> > > (AgentManager-Handler-13:null) (logid:) Ping from Routing host
> > > 1(srv-2-comp.aaaaaaaa.bbb)
> > >
> > > 2022-08-29 17:01:53,718 DEBUG [o.a.c.h.HAManagerImpl]
> > > (BackgroundTaskPollManager-4:ctx-b11cfa22) (logid:3163613f) HA health
> > check
> > > task is running...
> > > 2022-08-29 17:01:53,955 DEBUG [c.c.a.t.Request]
> > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
> > 1-5410793477308947403:
> > > Received: { Ans: , MgmtId: 90520740759984, via:
> > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
> > > 2022-08-29 17:01:54,025 DEBUG [c.c.a.t.Request]
> > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
> > 5-8959067033723473396:
> > > Received: { Ans: , MgmtId: 90520740759984, via:
> > > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
> > > 2022-08-29 17:01:54,096 DEBUG [c.c.a.t.Request]
> > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
> > 6-4833769775052102465:
> > > Received: { Ans: , MgmtId: 90520740759984, via:
> > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
> > > 2022-08-29 17:01:54,217 DEBUG [c.c.a.t.Request]
> > > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
> > 7-295548725546192473:
> > > Received: { Ans: , MgmtId: 90520740759984, via:
> > > 7(srv-5-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
> > > 2022-08-29 17:01:55,765 DEBUG [c.c.s.StatsCollector]
> > > (secstorage-1:ctx-7d9995d7) (logid:e388a8b8) Verifying image storage
> [1].
> > > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
> > > 2022-08-29 17:01:55,771 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > > (secstorage-1:ctx-7d9995d7) (logid:e388a8b8) Zone [1] is ready to
> launch
> > > secondary storage VM.
> > > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Zone 1 is ready to
> launch
> > > console proxy
> > > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Expand console proxy
> > standby
> > > capacity for zone xxxxxxxxxx
> > > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Allocating console proxy
> > > standby capacity for zone [1].
> > > 2022-08-29 17:01:56,034 INFO [c.c.c.ConsoleProxyManagerImpl]
> > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Found a stopped console
> > > proxy, starting it. Vm id : 85
> > > 2022-08-29 17:01:56,035 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) start parameter value of
> > > enterHardwareSetup == <very null> during processing of queued job
> > > 2022-08-29 17:01:56,044 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Sync job-4288 execution
> on
> > > object VmWorkJobQueue.85
> > > 2022-08-29 17:01:56,173 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (AsyncJobMgr-Heartbeat-1:ctx-e75826b1) (logid:e80fe659) Execute
> > sync-queue
> > > item: SyncQueueItemVO {id:2105, queueId: 2105, contentType: AsyncJob,
> > > contentId: 4288, lastProcessMsid: 90520740759984, lastprocessNumber: 1,
> > > lastProcessTime: Mon Aug 29 17:01:56 CEST 2022, created: Mon Aug 29
> > > 17:01:56 CEST 2022}
> > > 2022-08-29 17:01:56,174 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (AsyncJobMgr-Heartbeat-1:ctx-e75826b1) (logid:e80fe659) Schedule queued
> > > job-4288
> > > 2022-08-29 17:01:56,179 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:7e671dd0)
> > Add
> > > job-4288 into job monitoring
> > > 2022-08-29 17:01:56,186 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5)
> > > Executing AsyncJobVO: {id:4288, userId: 1, accountId: 1, instanceType:
> > > null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
> > >
> >
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAVXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
> > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> > > result: null, initMsid: 90520740759984, completeMsid: null,
> lastUpdated:
> > > null, lastPolled: null, created: Mon Aug 29 17:01:56 CEST 2022,
> removed:
> > > null}
> > > 2022-08-29 17:01:56,186 DEBUG [c.c.v.VmWorkJobDispatcher]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5)
> > Run
> > > VM work job: com.cloud.vm.VmWorkStart for VM 85, job origin: 3445
> > > 2022-08-29 17:01:56,187 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > (logid:ba10fec5) Execute VM work job:
> > >
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> > > 2022-08-29 17:01:56,189 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > (logid:ba10fec5) orchestrating VM start for 'v-85-VM'
> > > com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
> > > 2022-08-29 17:01:56,196 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
> > > transited from [Stopped] to [Starting] with event [StartRequested].
> VM's
> > > original host: null, new host: null, host before state transition: null
> > > 2022-08-29 17:01:56,196 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > (logid:ba10fec5) Successfully transitioned to start state for VM
> instance
> > > {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80",
> > > type="ConsoleProxy"} reservation id =
> > 0ab68202-d8be-469c-9b2a-87a55c2dc489
> > > 2022-08-29 17:01:56,200 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts: null
> > > 2022-08-29 17:01:56,202 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > (logid:ba10fec5) VM start attempt #1
> > > 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
> > > 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > (logid:ba10fec5) Trying to allocate a host and storage pools from dc:1,
> > > pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
> > > 1073741824
> > > 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
> > > 2022-08-29 17:01:56,212 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
> > > 2022-08-29 17:01:56,214 DEBUG [c.c.d.FirstFitPlanner]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > (logid:ba10fec5) Searching all possible resources under this Zone: 1
> > > 2022-08-29 17:01:56,215 DEBUG [c.c.d.FirstFitPlanner]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > (logid:ba10fec5) Listing clusters in order of aggregate capacity, that
> > have
> > > (at least one host with) enough CPU and RAM capacity under this Zone: 1
> > > 2022-08-29 17:01:56,218 DEBUG [c.c.d.FirstFitPlanner]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > (logid:ba10fec5) Removing from the clusterId list these clusters from
> > avoid
> > > set: []
> > > 2022-08-29 17:01:56,224 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
> > > 2022-08-29 17:01:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested with
> > > Custom[UEFI] Boot Type false
> > > 2022-08-29 17:01:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in dc: 1
> > > pod:1 cluster:1
> > > 2022-08-29 17:01:56,226 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4
> hosts
> > to
> > > check for allocation: [Host {"id": "5", "name":
> > "srv-4-comp.aaaaaaaa.bbb",
> > > "uuid": "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
> > > {"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb", "uuid":
> > > "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host {"id":
> > "6",
> > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id":
> > "1",
> > > "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> > > 2022-08-29 17:01:56,232 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for allocation
> > > after prioritization: [Host {"id": "5", "name":
> > "srv-4-comp.aaaaaaaa.bbb",
> > > "uuid": "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
> > > {"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb", "uuid":
> > > "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host {"id":
> > "6",
> > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id":
> > "1",
> > > "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> > > 2022-08-29 17:01:56,232 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for speed=500Mhz,
> > > Ram=1024 MB
> > > 2022-08-29 17:01:56,232 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
> > > srv-4-comp.aaaaaaaa.bbb, uuid: 134c625d-15ab-461a-a38d-36fb0b979b5d} is
> > KVM
> > > hypervisor type, no max guest limit check needed
> > > 2022-08-29 17:01:56,234 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu capability
> > > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed:
> 500
> > > 2022-08-29 17:01:56,234 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has
> enough
> > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> 1073741824 ,
> > > cpuOverprovisioningFactor: 1.0
> > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> > 127200
> > > and CPU after applying overprovisioning: 127200
> > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 , Requested
> > > CPU: 500
> > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
> > > 251009040384 , Requested RAM: (1.00 GB) 1073741824
> > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > > available
> > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> > host:
> > > 5, used: 11100, reserved: 0, actual total: 127200, total with
> > > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > > ,considerReservedCapacity?: true
> > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> > host:
> > > 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total: (250.27
> > GB)
> > > 268725780480; requested mem: (1.00 GB) 1073741824,
> alloc_from_last_host?:
> > > false , considerReservedCapacity?: true
> > > 2022-08-29 17:01:56,235 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> adding
> > to
> > > list: 5
> > > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
> > > srv-5-comp.aaaaaaaa.bbb, uuid: 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is
> > KVM
> > > hypervisor type, no max guest limit check needed
> > > 2022-08-29 17:01:56,238 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu capability
> > > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed:
> 500
> > > 2022-08-29 17:01:56,238 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has
> enough
> > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> 1073741824 ,
> > > cpuOverprovisioningFactor: 1.0
> > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> > 127200
> > > and CPU after applying overprovisioning: 127200
> > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 , Requested
> > CPU:
> > > 500
> > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
> > > 189373743104 , Requested RAM: (1.00 GB) 1073741824
> > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > > available
> > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> > host:
> > > 7, used: 90100, reserved: 0, actual total: 127200, total with
> > > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > > ,considerReservedCapacity?: true
> > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> > host:
> > > 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total: (250.27
> > GB)
> > > 268725780480; requested mem: (1.00 GB) 1073741824,
> alloc_from_last_host?:
> > > false , considerReservedCapacity?: true
> > > 2022-08-29 17:01:56,240 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> adding
> > to
> > > list: 7
> > > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
> > > srv-3-comp.aaaaaaaa.bbb, uuid: 6858713b-7e17-46d8-854d-24a13f9c7001} is
> > KVM
> > > hypervisor type, no max guest limit check needed
> > > 2022-08-29 17:01:56,242 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu capability
> > > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed:
> 500
> > > 2022-08-29 17:01:56,242 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has
> enough
> > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> 1073741824 ,
> > > cpuOverprovisioningFactor: 1.0
> > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> > 127200
> > > and CPU after applying overprovisioning: 127200
> > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 , Requested
> > CPU:
> > > 500
> > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
> > > 204032835584 , Requested RAM: (1.00 GB) 1073741824
> > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > > available
> > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> > host:
> > > 6, used: 42900, reserved: 0, actual total: 127200, total with
> > > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > > ,considerReservedCapacity?: true
> > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> > host:
> > > 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total: (250.27
> > GB)
> > > 268725780480; requested mem: (1.00 GB) 1073741824,
> alloc_from_last_host?:
> > > false , considerReservedCapacity?: true
> > > 2022-08-29 17:01:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> adding
> > to
> > > list: 6
> > > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
> > > srv-2-comp.aaaaaaaa.bbb, uuid: 883f7f62-55f2-42f4-8242-16b6847c4204} is
> > KVM
> > > hypervisor type, no max guest limit check needed
> > > 2022-08-29 17:01:56,246 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu capability
> > > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed:
> 500
> > > 2022-08-29 17:01:56,246 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has
> enough
> > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> 1073741824 ,
> > > cpuOverprovisioningFactor: 1.0
> > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> > 127200
> > > and CPU after applying overprovisioning: 127200
> > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 , Requested
> > CPU:
> > > 500
> > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
> > > 229802639360 , Requested RAM: (1.00 GB) 1073741824
> > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > > available
> > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> > host:
> > > 1, used: 48200, reserved: 0, actual total: 127200, total with
> > > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > > ,considerReservedCapacity?: true
> > > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> > host:
> > > 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total: (250.27
> > GB)
> > > 268725780480; requested mem: (1.00 GB) 1073741824,
> alloc_from_last_host?:
> > > false , considerReservedCapacity?: true
> > > 2022-08-29 17:01:56,247 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> adding
> > to
> > > list: 1
> > > 2022-08-29 17:01:56,247 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator returning 4
> > > suitable hosts
> > > 2022-08-29 17:01:56,258 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
> > > transited from [Starting] to [Stopped] with event [OperationFailed].
> VM's
> > > original host: null, new host: null, host before state transition: null
> > > 2022-08-29 17:01:56,258 WARN [c.c.v.VirtualMachineManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > (logid:ba10fec5) Unable to orchestrate start VM instance {id: "85",
> name:
> > > "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
> > > type="ConsoleProxy"} due to [Unable to create deployment, no usable
> > volumes
> > > found for the VM: 85].
> > > com.cloud.utils.exception.CloudRuntimeException: Unable to create
> > > deployment, no usable volumes found for the VM: 85
> > > at
> > >
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
> > > at
> > >
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
> > > at
> > >
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
> > > at
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
> > > at
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
> > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> Source)
> > > at
> > >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > > at
> > >
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > > at
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > > at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > > at
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > > at
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > > at
> > >
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > > at
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > at
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > at java.base/java.lang.Thread.run(Thread.java:829)
> > > 2022-08-29 17:01:56,259 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > (logid:ba10fec5) Done executing VM work job:
> > >
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> > > 2022-08-29 17:01:56,260 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > (logid:ba10fec5) Complete async job-4288, jobStatus: FAILED,
> resultCode:
> > 0,
> > > result:
> > >
> >
> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODUiLCBuYW1lOiAidi04NS1WTSIsIHV1aWQ6ICIxMTBkODYyMC1jMDI5LTRjYTktOTA4NC0yMTAwOTk3MjVhODAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODVdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
> > > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > (logid:ba10fec5) Publish async job-4288 complete on message bus
> > > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > (logid:ba10fec5) Wake up jobs related to job-4288
> > > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > (logid:ba10fec5) Update db status for job-4288
> > > 2022-08-29 17:01:56,262 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > > (logid:ba10fec5) Wake up jobs joined with job-4288 and disjoin all
> > subjobs
> > > created from job- 4288
> > > 2022-08-29 17:01:56,270 DEBUG [c.c.v.VmWorkJobDispatcher]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5)
> > > Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 85, job
> > > origin: 3445
> > > 2022-08-29 17:01:56,270 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5)
> > > Done executing com.cloud.vm.VmWorkStart for job-4288
> > > 2022-08-29 17:01:56,272 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5)
> > > Remove job-4288 from job monitoring
> > > 2022-08-29 17:01:56,277 WARN [c.c.c.ConsoleProxyManagerImpl]
> > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Unable to allocate
> console
> > > proxy standby capacity for zone [1] due to [Unable to orchestrate start
> > VM
> > > instance {id: "85", name: "v-85-VM", uuid:
> > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> > [Unable
> > > to create deployment, no usable volumes found for the VM: 85].].
> > > com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate
> > > start VM instance {id: "85", name: "v-85-VM", uuid:
> > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> > [Unable
> > > to create deployment, no usable volumes found for the VM: 85].
> > > at
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> Source)
> > > at
> > >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > > at
> > >
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > > at
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > > at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > > at
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > > at
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > > at
> > >
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > > at
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > at
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > at java.base/java.lang.Thread.run(Thread.java:829)
> > > 2022-08-29 17:01:56,277 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) received console proxy
> > alert
> > > 2022-08-29 17:01:56,278 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Console proxy creation
> > > failure, zone: xxxxxxxxxx
> > > 2022-08-29 17:01:56,279 WARN [c.c.a.AlertManagerImpl]
> > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) alertType=[10]
> > > dataCenterId=[1] podId=[null] clusterId=[null] message=[Console proxy
> > > creation failure. zone: xxxxxxxxxx, error details: Unable to
> orchestrate
> > > start VM instance {id: "85", name: "v-85-VM", uuid:
> > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> > [Unable
> > > to create deployment, no usable volumes found for the VM: 85].].
> > > 2022-08-29 17:01:56,284 WARN [c.c.a.AlertManagerImpl]
> > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) No recipients set in
> > global
> > > setting 'alert.email.addresses', skipping sending alert with subject
> > > [Console proxy creation failure. zone: xxxxxxxxxx, error details:
> Unable
> > to
> > > orchestrate start VM instance {id: "85", name: "v-85-VM", uuid:
> > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> > [Unable
> > > to create deployment, no usable volumes found for the VM: 85].] and
> > content
> > > [Console proxy creation failure (zone xxxxxxxxxx)].
> > > 2022-08-29 17:01:56,284 WARN [c.c.v.SystemVmLoadScanner]
> > > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Unexpected exception
> > Unable
> > > to orchestrate start VM instance {id: "85", name: "v-85-VM", uuid:
> > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> > [Unable
> > > to create deployment, no usable volumes found for the VM: 85].
> > > com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate
> > > start VM instance {id: "85", name: "v-85-VM", uuid:
> > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> > [Unable
> > > to create deployment, no usable volumes found for the VM: 85].
> > > at
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> Source)
> > > at
> > >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > > at
> > >
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > > at
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > > at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > > at
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > > at
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > > at
> > >
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > > at
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > at
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > at java.base/java.lang.Thread.run(Thread.java:829)
> > > 2022-08-29 17:01:57,721 DEBUG [o.a.c.h.HAManagerImpl]
> > > (BackgroundTaskPollManager-4:ctx-9528226f) (logid:4512ff9b) HA health
> > check
> > > task is running...
> > > 2022-08-29 17:01:59,504 DEBUG [c.c.s.StatsCollector]
> > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) StorageCollector is
> > > running...
> > > 2022-08-29 17:01:59,518 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
> > getCommandHostDelegation:
> > > class com.cloud.agent.api.GetStorageStatsCommand
> > > 2022-08-29 17:01:59,518 DEBUG [c.c.h.XenServerGuru]
> > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning the
> > > default host to execute commands because the command is not of Copy
> type.
> > > 2022-08-29 17:01:59,571 DEBUG [c.c.a.t.Request]
> > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
> > > 11-7523826127475836583: Received: { Ans: , MgmtId: 90520740759984,
> via:
> > > 11(s-32-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> > > 2022-08-29 17:01:59,574 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
> > getCommandHostDelegation:
> > > class com.cloud.agent.api.GetStorageStatsCommand
> > > 2022-08-29 17:01:59,574 DEBUG [c.c.h.XenServerGuru]
> > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning the
> > > default host to execute commands because the command is not of Copy
> type.
> > > 2022-08-29 17:01:59,615 DEBUG [c.c.a.t.Request]
> > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
> > 5-8959067033723473397:
> > > Received: { Ans: , MgmtId: 90520740759984, via:
> > > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetStorageStatsAnswer
> > } }
> > > 2022-08-29 17:01:59,624 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
> > getCommandHostDelegation:
> > > class com.cloud.agent.api.GetStorageStatsCommand
> > > 2022-08-29 17:01:59,624 DEBUG [c.c.h.XenServerGuru]
> > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning the
> > > default host to execute commands because the command is not of Copy
> type.
> > > 2022-08-29 17:01:59,653 DEBUG [c.c.a.t.Request]
> > > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
> > 6-4833769775052102466:
> > > Received: { Ans: , MgmtId: 90520740759984, via:
> > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetStorageStatsAnswer
> > } }
> > > 2022-08-29 17:01:59,979 DEBUG [c.c.a.m.AgentManagerImpl]
> > > (AgentManager-Handler-12:null) (logid:) Ping from Routing host
> > > 5(srv-4-comp.aaaaaaaa.bbb)
> > >
> > > 2022-08-29 17:02:00,269 DEBUG [c.c.s.StatsCollector]
> > > (StatsCollector-5:ctx-d2b6ffd3) (logid:95201678) DbCollector is
> > running...
> > > 2022-08-29 17:02:00,443 DEBUG
> > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 2 routers
> to
> > > update status.
> > > 2022-08-29 17:02:00,449 DEBUG
> > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 1 VPC's to
> > > update Redundant State.
> > > 2022-08-29 17:02:00,450 DEBUG
> > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 0 networks
> to
> > > update RvR status.
> > > 2022-08-29 17:02:00,457 DEBUG [c.c.a.t.Request]
> > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
> > > 6-4833769775052102467: Sending { Cmd , MgmtId: 90520740759984, via:
> > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
> > router.name
> >
> ":"r-79-VM","router.ip":"169.254.144.198"},"wait":"30","bypassHostMaintenance":"false"}}]
> > > }
> > > 2022-08-29 17:02:00,599 DEBUG [c.c.a.t.Request]
> > > (AgentManager-Handler-7:null) (logid:) Seq 6-4833769775052102467:
> > > Processing: { Ans: , MgmtId: 90520740759984, via: 6, Ver: v1, Flags:
> 10,
> > >
> >
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"PRIMARY","result":"true","details":"Status:
> > > PRIMARY
> > > ","wait":"0","bypassHostMaintenance":"false"}}] }
> > > 2022-08-29 17:02:00,599 DEBUG [c.c.a.t.Request]
> > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
> > > 6-4833769775052102467: Received: { Ans: , MgmtId: 90520740759984, via:
> > > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer } }
> > > 2022-08-29 17:02:00,599 DEBUG [c.c.a.m.AgentManagerImpl]
> > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Details
> > > from executing class com.cloud.agent.api.CheckRouterCommand: Status:
> > PRIMARY
> > >
> > > 2022-08-29 17:02:00,612 DEBUG [c.c.a.t.Request]
> > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
> > > 1-5410793477308947404: Sending { Cmd , MgmtId: 90520740759984, via:
> > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> > > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
> > router.name
> >
> ":"r-80-VM","router.ip":"169.254.217.200"},"wait":"30","bypassHostMaintenance":"false"}}]
> > > }
> > > 2022-08-29 17:02:00,748 DEBUG [c.c.a.m.AgentManagerImpl]
> > > (AgentManager-Handler-4:null) (logid:) Ping from 11(s-32-VM)
> > > 2022-08-29 17:02:00,750 DEBUG [c.c.a.t.Request]
> > > (AgentManager-Handler-1:null) (logid:) Seq 1-5410793477308947404:
> > > Processing: { Ans: , MgmtId: 90520740759984, via: 1, Ver: v1, Flags:
> 10,
> > >
> >
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"BACKUP","result":"true","details":"Status:
> > > BACKUP
> > > ","wait":"0","bypassHostMaintenance":"false"}}] }
> > > 2022-08-29 17:02:00,750 DEBUG [c.c.a.t.Request]
> > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
> > > 1-5410793477308947404: Received: { Ans: , MgmtId: 90520740759984, via:
> > > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer } }
> > > 2022-08-29 17:02:00,751 DEBUG [c.c.a.m.AgentManagerImpl]
> > > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Details
> > > from executing class com.cloud.agent.api.CheckRouterCommand: Status:
> > BACKUP
> > >
> > > 2022-08-29 17:02:01,362 DEBUG [c.c.s.StatsCollector]
> > > (Cluster-Worker-839:ctx-e097a8ab) (logid:3deacf65) StatusUpdate from
> > > 90520740759984, json:
> > >
> >
> {"managementServerHostId":1,"managementServerHostUuid":"5615ca5f-0716-4da1-93ec-31f79fd19bbe","collectionTime":"Aug
> > > 29, 2022, 5:02:01
> > >
> >
> PM","sessions":1,"cpuUtilization":0.0,"totalJvmMemoryBytes":324534272,"freeJvmMemoryBytes":68595424,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":103760388,"jvmStartTime":1661681560910,"availableProcessors":6,"loadAverage":0.02,"totalInit":139788288,"totalUsed":474791104,"totalCommitted":557662208,"pid":1522332,"jvmName":"1522332@vm-1-cs-mgmt.aaaaaaaa.bbb
> >
> ","jvmVendor":"Ubuntu","jvmVersion":"11.0.16+8-post-Ubuntu-0ubuntu120.04","osDistribution":"Ubuntu
> > > 20.04.5
> > >
> >
> LTS","agentCount":5,"heapMemoryUsed":256775312,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":13,"threadsRunnableCount":15,"threadsTerminatedCount":0,"threadsTotalCount":420,"threadsWaitingCount":331,"systemMemoryTotal":8343236608,"systemMemoryFree":2623193088,"systemMemoryUsed":814808,"systemMemoryVirtualSize":7494184960,"logInfo":"/var/log/cloudstack/management/management-server.log
> > > using: 74M\non disk /dev/vda2 mounted on / (14%
> > >
> >
> full)","systemTotalCpuCycles":15870.180000000002,"systemLoadAverages":[0.02,0.03,0.0],"systemCyclesUsage":[4116569,2615101,2716460424],"dbLocal":false,"usageLocal":false,"systemBootTime":"Jul
> > > 8, 2022, 2:35:55 AM","kernelVersion":"5.4.0-121-generic"}
> > > 2022-08-29 17:02:01,723 DEBUG [o.a.c.h.HAManagerImpl]
> > > (BackgroundTaskPollManager-4:ctx-7abb1b5d) (logid:2320950d) HA health
> > check
> > > task is running...
> > > 2022-08-29 17:02:05,417 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > (Vm-Operations-Cleanup-1:ctx-e741e208) (logid:3f5129e0) VM Operation
> > Thread
> > > Running
> > > 2022-08-29 17:02:05,725 DEBUG [o.a.c.h.HAManagerImpl]
> > > (BackgroundTaskPollManager-4:ctx-8e7f6766) (logid:710cb3b2) HA health
> > check
> > > task is running...
> > > 2022-08-29 17:02:08,626 DEBUG [c.c.a.m.AgentManagerImpl]
> > > (AgentManager-Handler-11:null) (logid:) Ping from Routing host
> > > 6(srv-3-comp.aaaaaaaa.bbb)
> > >
> > > 2022-08-29 17:02:09,727 DEBUG [o.a.c.h.HAManagerImpl]
> > > (BackgroundTaskPollManager-4:ctx-26a5ff07) (logid:09b3c89a) HA health
> > check
> > > task is running...
> > > 2022-08-29 17:02:13,729 DEBUG [o.a.c.h.HAManagerImpl]
> > > (BackgroundTaskPollManager-4:ctx-b4d4e374) (logid:9048c7a7) HA health
> > check
> > > task is running...
> > > 2022-08-29 17:02:17,731 DEBUG [o.a.c.h.HAManagerImpl]
> > > (BackgroundTaskPollManager-4:ctx-3db3d208) (logid:7bb73f7b) HA health
> > check
> > > task is running...
> > > 2022-08-29 17:02:21,734 DEBUG [o.a.c.h.HAManagerImpl]
> > > (BackgroundTaskPollManager-1:ctx-fca1d248) (logid:1ed5381a) HA health
> > check
> > > task is running...
> > > 2022-08-29 17:02:25,737 DEBUG [o.a.c.h.HAManagerImpl]
> > > (BackgroundTaskPollManager-1:ctx-984e2382) (logid:e6c3d724) HA health
> > check
> > > task is running...
> > > 2022-08-29 17:02:25,759 DEBUG [c.c.s.StatsCollector]
> > > (secstorage-1:ctx-64c610ed) (logid:6603cd38) Verifying image storage
> [1].
> > > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
> > > 2022-08-29 17:02:25,763 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > > (secstorage-1:ctx-64c610ed) (logid:6603cd38) Zone [1] is ready to
> launch
> > > secondary storage VM.
> > > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Zone 1 is ready to
> launch
> > > console proxy
> > > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Expand console proxy
> > standby
> > > capacity for zone xxxxxxxxxx
> > > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Allocating console proxy
> > > standby capacity for zone [1].
> > > 2022-08-29 17:02:26,030 INFO [c.c.c.ConsoleProxyManagerImpl]
> > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Found a stopped console
> > > proxy, starting it. Vm id : 85
> > > 2022-08-29 17:02:26,031 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) start parameter value of
> > > enterHardwareSetup == <very null> during processing of queued job
> > > 2022-08-29 17:02:26,037 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Sync job-4289 execution
> on
> > > object VmWorkJobQueue.85
> > > 2022-08-29 17:02:26,167 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (AsyncJobMgr-Heartbeat-1:ctx-bba6b89d) (logid:f9aed0eb) Execute
> > sync-queue
> > > item: SyncQueueItemVO {id:2106, queueId: 2105, contentType: AsyncJob,
> > > contentId: 4289, lastProcessMsid: 90520740759984, lastprocessNumber: 2,
> > > lastProcessTime: Mon Aug 29 17:02:26 CEST 2022, created: Mon Aug 29
> > > 17:02:26 CEST 2022}
> > > 2022-08-29 17:02:26,167 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (AsyncJobMgr-Heartbeat-1:ctx-bba6b89d) (logid:f9aed0eb) Schedule queued
> > > job-4289
> > > 2022-08-29 17:02:26,172 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:f919f9b3)
> > Add
> > > job-4289 into job monitoring
> > > 2022-08-29 17:02:26,181 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5)
> > > Executing AsyncJobVO: {id:4289, userId: 1, accountId: 1, instanceType:
> > > null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
> > >
> >
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAVXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
> > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> > > result: null, initMsid: 90520740759984, completeMsid: null,
> lastUpdated:
> > > null, lastPolled: null, created: Mon Aug 29 17:02:26 CEST 2022,
> removed:
> > > null}
> > > 2022-08-29 17:02:26,181 DEBUG [c.c.v.VmWorkJobDispatcher]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5)
> > Run
> > > VM work job: com.cloud.vm.VmWorkStart for VM 85, job origin: 3445
> > > 2022-08-29 17:02:26,183 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > (logid:ba10fec5) Execute VM work job:
> > >
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> > > 2022-08-29 17:02:26,184 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > (logid:ba10fec5) orchestrating VM start for 'v-85-VM'
> > > com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
> > > 2022-08-29 17:02:26,192 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
> > > transited from [Stopped] to [Starting] with event [StartRequested].
> VM's
> > > original host: null, new host: null, host before state transition: null
> > > 2022-08-29 17:02:26,192 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > (logid:ba10fec5) Successfully transitioned to start state for VM
> instance
> > > {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80",
> > > type="ConsoleProxy"} reservation id =
> > 24bf955f-96d8-4f4f-ae33-2cae2f3e0425
> > > 2022-08-29 17:02:26,198 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts: null
> > > 2022-08-29 17:02:26,200 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > (logid:ba10fec5) VM start attempt #1
> > > 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
> > > 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > (logid:ba10fec5) Trying to allocate a host and storage pools from dc:1,
> > > pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
> > > 1073741824
> > > 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
> > > 2022-08-29 17:02:26,211 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
> > > 2022-08-29 17:02:26,212 DEBUG [c.c.d.FirstFitPlanner]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > (logid:ba10fec5) Searching all possible resources under this Zone: 1
> > > 2022-08-29 17:02:26,213 DEBUG [c.c.d.FirstFitPlanner]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > (logid:ba10fec5) Listing clusters in order of aggregate capacity, that
> > have
> > > (at least one host with) enough CPU and RAM capacity under this Zone: 1
> > > 2022-08-29 17:02:26,216 DEBUG [c.c.d.FirstFitPlanner]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > (logid:ba10fec5) Removing from the clusterId list these clusters from
> > avoid
> > > set: []
> > > 2022-08-29 17:02:26,221 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
> > > 2022-08-29 17:02:26,222 INFO [c.c.a.m.a.i.FirstFitAllocator]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested with
> > > Custom[UEFI] Boot Type false
> > > 2022-08-29 17:02:26,223 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in dc: 1
> > > pod:1 cluster:1
> > > 2022-08-29 17:02:26,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4
> hosts
> > to
> > > check for allocation: [Host {"id": "7", "name":
> > "srv-5-comp.aaaaaaaa.bbb",
> > > "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
> > > {"id": "1", "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}, Host {"id":
> > "6",
> > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id":
> > "5",
> > > "name": "srv-4-comp.aaaaaaaa.bbb", "uuid":
> > > "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}]
> > > 2022-08-29 17:02:26,231 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for allocation
> > > after prioritization: [Host {"id": "7", "name":
> > "srv-5-comp.aaaaaaaa.bbb",
> > > "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
> > > {"id": "1", "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> > > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}, Host {"id":
> > "6",
> > > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> > > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id":
> > "5",
> > > "name": "srv-4-comp.aaaaaaaa.bbb", "uuid":
> > > "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}]
> > > 2022-08-29 17:02:26,231 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for speed=500Mhz,
> > > Ram=1024 MB
> > > 2022-08-29 17:02:26,231 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
> > > srv-5-comp.aaaaaaaa.bbb, uuid: 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is
> > KVM
> > > hypervisor type, no max guest limit check needed
> > > 2022-08-29 17:02:26,234 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu capability
> > > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed:
> 500
> > > 2022-08-29 17:02:26,234 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has
> enough
> > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> 1073741824 ,
> > > cpuOverprovisioningFactor: 1.0
> > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> > 127200
> > > and CPU after applying overprovisioning: 127200
> > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 , Requested
> > CPU:
> > > 500
> > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
> > > 189373743104 , Requested RAM: (1.00 GB) 1073741824
> > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > > available
> > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> > host:
> > > 7, used: 90100, reserved: 0, actual total: 127200, total with
> > > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > > ,considerReservedCapacity?: true
> > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> > host:
> > > 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total: (250.27
> > GB)
> > > 268725780480; requested mem: (1.00 GB) 1073741824,
> alloc_from_last_host?:
> > > false , considerReservedCapacity?: true
> > > 2022-08-29 17:02:26,236 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> adding
> > to
> > > list: 7
> > > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
> > > srv-2-comp.aaaaaaaa.bbb, uuid: 883f7f62-55f2-42f4-8242-16b6847c4204} is
> > KVM
> > > hypervisor type, no max guest limit check needed
> > > 2022-08-29 17:02:26,239 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu capability
> > > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed:
> 500
> > > 2022-08-29 17:02:26,239 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has
> enough
> > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> 1073741824 ,
> > > cpuOverprovisioningFactor: 1.0
> > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> > 127200
> > > and CPU after applying overprovisioning: 127200
> > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 , Requested
> > CPU:
> > > 500
> > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
> > > 229802639360 , Requested RAM: (1.00 GB) 1073741824
> > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > > available
> > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> > host:
> > > 1, used: 48200, reserved: 0, actual total: 127200, total with
> > > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > > ,considerReservedCapacity?: true
> > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> > host:
> > > 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total: (250.27
> > GB)
> > > 268725780480; requested mem: (1.00 GB) 1073741824,
> alloc_from_last_host?:
> > > false , considerReservedCapacity?: true
> > > 2022-08-29 17:02:26,240 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> adding
> > to
> > > list: 1
> > > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
> > > srv-3-comp.aaaaaaaa.bbb, uuid: 6858713b-7e17-46d8-854d-24a13f9c7001} is
> > KVM
> > > hypervisor type, no max guest limit check needed
> > > 2022-08-29 17:02:26,243 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu capability
> > > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed:
> 500
> > > 2022-08-29 17:02:26,243 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has
> enough
> > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> 1073741824 ,
> > > cpuOverprovisioningFactor: 1.0
> > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> > 127200
> > > and CPU after applying overprovisioning: 127200
> > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 , Requested
> > CPU:
> > > 500
> > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
> > > 204032835584 , Requested RAM: (1.00 GB) 1073741824
> > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > > available
> > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> > host:
> > > 6, used: 42900, reserved: 0, actual total: 127200, total with
> > > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > > ,considerReservedCapacity?: true
> > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> > host:
> > > 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total: (250.27
> > GB)
> > > 268725780480; requested mem: (1.00 GB) 1073741824,
> alloc_from_last_host?:
> > > false , considerReservedCapacity?: true
> > > 2022-08-29 17:02:26,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> adding
> > to
> > > list: 6
> > > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
> > > srv-4-comp.aaaaaaaa.bbb, uuid: 134c625d-15ab-461a-a38d-36fb0b979b5d} is
> > KVM
> > > hypervisor type, no max guest limit check needed
> > > 2022-08-29 17:02:26,247 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu capability
> > > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed:
> 500
> > > 2022-08-29 17:02:26,247 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has
> enough
> > > capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> 1073741824 ,
> > > cpuOverprovisioningFactor: 1.0
> > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> > 127200
> > > and CPU after applying overprovisioning: 127200
> > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 , Requested
> > > CPU: 500
> > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
> > > 251009040384 , Requested RAM: (1.00 GB) 1073741824
> > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > > available
> > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> > host:
> > > 5, used: 11100, reserved: 0, actual total: 127200, total with
> > > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > > ,considerReservedCapacity?: true
> > > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> > host:
> > > 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total: (250.27
> > GB)
> > > 268725780480; requested mem: (1.00 GB) 1073741824,
> alloc_from_last_host?:
> > > false , considerReservedCapacity?: true
> > > 2022-08-29 17:02:26,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> adding
> > to
> > > list: 5
> > > 2022-08-29 17:02:26,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > > FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator returning 4
> > > suitable hosts
> > > 2022-08-29 17:02:26,260 DEBUG [c.c.c.CapacityManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
> > > transited from [Starting] to [Stopped] with event [OperationFailed].
> VM's
> > > original host: null, new host: null, host before state transition: null
> > > 2022-08-29 17:02:26,260 WARN [c.c.v.VirtualMachineManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > (logid:ba10fec5) Unable to orchestrate start VM instance {id: "85",
> name:
> > > "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
> > > type="ConsoleProxy"} due to [Unable to create deployment, no usable
> > volumes
> > > found for the VM: 85].
> > > com.cloud.utils.exception.CloudRuntimeException: Unable to create
> > > deployment, no usable volumes found for the VM: 85
> > > at
> > >
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
> > > at
> > >
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
> > > at
> > >
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
> > > at
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
> > > at
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
> > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> Source)
> > > at
> > >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > > at
> > >
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > > at
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > > at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > > at
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > > at
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > > at
> > >
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > > at
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > at
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > at java.base/java.lang.Thread.run(Thread.java:829)
> > > 2022-08-29 17:02:26,261 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > (logid:ba10fec5) Done executing VM work job:
> > >
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> > > 2022-08-29 17:02:26,262 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > (logid:ba10fec5) Complete async job-4289, jobStatus: FAILED,
> resultCode:
> > 0,
> > > result:
> > >
> >
> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODUiLCBuYW1lOiAidi04NS1WTSIsIHV1aWQ6ICIxMTBkODYyMC1jMDI5LTRjYTktOTA4NC0yMTAwOTk3MjVhODAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODVdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
> > > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > (logid:ba10fec5) Publish async job-4289 complete on message bus
> > > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > (logid:ba10fec5) Wake up jobs related to job-4289
> > > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > (logid:ba10fec5) Update db status for job-4289
> > > 2022-08-29 17:02:26,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > > (logid:ba10fec5) Wake up jobs joined with job-4289 and disjoin all
> > subjobs
> > > created from job- 4289
> > > 2022-08-29 17:02:26,269 DEBUG [c.c.v.VmWorkJobDispatcher]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5)
> > > Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 85, job
> > > origin: 3445
> > > 2022-08-29 17:02:26,270 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5)
> > > Done executing com.cloud.vm.VmWorkStart for job-4289
> > > 2022-08-29 17:02:26,271 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5)
> > > Remove job-4289 from job monitoring
> > > 2022-08-29 17:02:26,277 WARN [c.c.c.ConsoleProxyManagerImpl]
> > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Unable to allocate
> console
> > > proxy standby capacity for zone [1] due to [Unable to orchestrate start
> > VM
> > > instance {id: "85", name: "v-85-VM", uuid:
> > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> > [Unable
> > > to create deployment, no usable volumes found for the VM: 85].].
> > > com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate
> > > start VM instance {id: "85", name: "v-85-VM", uuid:
> > > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> > [Unable
> > > to create deployment, no usable volumes found for the VM: 85].
> > > at
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> > > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> Source)
> > > at
> > >
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > > at
> > >
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > > at
> > >
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > > at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > > at
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > > at
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > > at
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > > at
> > >
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > > at
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > > at
> > >
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > > at java.base/java.lang.Thread.run(Thread.java:829)
> > > 2022-08-29 17:02:26,277 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) received console proxy
> > alert
> > >
> > >
> > >
> > > Am Mo., 29. Aug. 2022 um 16:30 Uhr schrieb Slavka Peleva <
> > > slavkap@storpool.com>:
> > >
> > >> Thanks, Chris, for the information!
> > >>
> > >> Are you using custom offerings for the system VMs? Can you share more
> > >> info if yes?
> > >>
> > >> About the log, it's not the complete log from where the deployment
> > >> started. The deployment begins a bit earlier than what you shared. Can
> > you
> > >> search the management log for `Sync job-3539 execution on object
> > >> VmWorkJobQueue` to get the `logid` for the volume creation (coloured
> in
> > >> blue)? And if possible, share the logs with the found log ID
> > >>
> > >> Here is an example from my logs:
> > >>
> > >> 2022-08-29 17:07:49,114 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > >> (consoleproxy-1:ctx-a952a3dc) (logid:89157bb9) Sync job-60484
> execution
> > >> on object VmWorkJobQueue.4904
> > >>
> > >> 2022-08-29 17:07:50,816 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> > >> (Work-Job-Executor-5:ctx-56172581 job-60479/job-60484 ctx-a900986d)
> > >> (logid:95d750ae) Execute VM work job:
> > >>
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":4904,"handlerName":"VirtualMachineManagerImpl"}
> > >> 2022-08-29 17:07:50,817 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > >> (Work-Job-Executor-5:ctx-56172581 job-60479/job-60484 ctx-a900986d)
> > >> (logid:95d750ae) orchestrating VM start for 'v-4904-VM'
> > >> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
> > >>
> > >>
> > >> On Mon, Aug 29, 2022 at 2:29 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
> > >>
> > >>> Hi Slavka,
> > >>>
> > >>> thats what i did after your suggestion / informatin and where the
> error
> > >>> occured.
> > >>> Deleted / Destroyed the consoleproxy and waited till recreation.
> Which
> > >>> didn't worked throwing the error i pasted above.
> > >>> Sorry for beeing so unspecific.
> > >>>
> > >>> Regards,
> > >>> Chris
> > >>>
> > >>> Am Mo., 29. Aug. 2022 um 12:46 Uhr schrieb Slavka Peleva <
> > >>> slavkap@storpool.com>:
> > >>>
> > >>>> Hi Chris,
> > >>>>
> > >>>> I mean to recreate the system VMs by destroying them.
> > >>>>
> > >>>> On Mon, Aug 29, 2022 at 12:38 PM Vash_X@gmx.de <Va...@gmx.de>
> wrote:
> > >>>>
> > >>>>> Hi Slavka,
> > >>>>>
> > >>>>> didn't tried to restart the SystemVMs. Reading through the docs i
> had
> > >>>>> the impression that "live-patching" would be enough for this....
> > >>>>> Anyway - I tried my luck with the console proxy.
> > >>>>>
> > >>>>> Which failed utterly. Not starting as it isn't finding any useable
> > >>>>> storage pool. - logs at the end of the mail.
> > >>>>>
> > >>>>> I tried deploying a VM using the same storage pool which worked
> > >>>>> wonderfully.
> > >>>>> There i noticed that all my custome storage-offerings are unuseable
> > at
> > >>>>> the moment. They are not accepting any custome storage sizes
> > anymore...
> > >>>>> Throwing failures "VM Creation failed. Volume size: -1GB is out of
> > allowed
> > >>>>> tange. Max:3072 Min:1"...
> > >>>>> So this is also currently broken....
> > >>>>>
> > >>>>> 2022-08-29 10:51:56,189 DEBUG [c.c.v.VmWorkJobDispatcher]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
> > (logid:ba10fec5) Run
> > >>>>> VM work job: com.cloud.vm.VmWorkStart for VM 83, job origin: 3445
> > >>>>> 2022-08-29 10:51:56,191 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> > >>>>> (logid:ba10fec5) Execute VM work job:
> > >>>>>
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
> > >>>>> 2022-08-29 10:51:56,192 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> > >>>>> (logid:ba10fec5) orchestrating VM start for 'v-83-VM'
> > >>>>> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
> > >>>>> 2022-08-29 10:51:56,198 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> > >>>>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
> > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} state
> > >>>>> transited from [Stopped] to [Starting] with event [StartRequested].
> > VM's
> > >>>>> original host: null, new host: null, host before state transition:
> > null
> > >>>>> 2022-08-29 10:51:56,198 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> > >>>>> (logid:ba10fec5) Successfully transitioned to start state for VM
> > instance
> > >>>>> {id: "83", name: "v-83-VM", uuid:
> > "b1d1f9c6-eafe-4527-9335-e649c646aab0",
> > >>>>> type="ConsoleProxy"} reservation id =
> > 6641ccc0-bcef-4b1a-a1f7-8dd8000626ad
> > >>>>> 2022-08-29 10:51:56,203 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> > >>>>> (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts:
> > null
> > >>>>> 2022-08-29 10:51:56,206 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> > >>>>> (logid:ba10fec5) VM start attempt #1
> > >>>>> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> > >>>>> (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
> > >>>>> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> > >>>>> (logid:ba10fec5) Trying to allocate a host and storage pools from
> > dc:1,
> > >>>>> pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
> > >>>>> 1073741824
> > >>>>> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> > >>>>> (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
> > >>>>> 2022-08-29 10:51:56,214 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> > >>>>> (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
> > >>>>> 2022-08-29 10:51:56,215 DEBUG [c.c.d.FirstFitPlanner]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> > >>>>> (logid:ba10fec5) Searching all possible resources under this Zone:
> 1
> > >>>>> 2022-08-29 10:51:56,216 DEBUG [c.c.d.FirstFitPlanner]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> > >>>>> (logid:ba10fec5) Listing clusters in order of aggregate capacity,
> > that have
> > >>>>> (at least one host with) enough CPU and RAM capacity under this
> > Zone: 1
> > >>>>> 2022-08-29 10:51:56,219 DEBUG [c.c.d.FirstFitPlanner]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> > >>>>> (logid:ba10fec5) Removing from the clusterId list these clusters
> > from avoid
> > >>>>> set: []
> > >>>>> 2022-08-29 10:51:56,224 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> > >>>>> (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
> > >>>>> 2022-08-29 10:51:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested
> > with
> > >>>>> Custom[UEFI] Boot Type false
> > >>>>> 2022-08-29 10:51:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in
> dc: 1
> > >>>>> pod:1 cluster:1
> > >>>>> 2022-08-29 10:51:56,227 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4
> > hosts to
> > >>>>> check for allocation: [Host {"id": "7", "name":
> > "srv-5-comp.blabla.blub",
> > >>>>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"},
> > Host
> > >>>>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
> > >>>>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
> > {"id": "6",
> > >>>>> "name": "srv-3-comp.blabla.blub", "uuid":
> > >>>>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
> > {"id": "1",
> > >>>>> "name": "srv-2-comp.blabla.blub", "uuid":
> > >>>>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> > >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for
> > allocation
> > >>>>> after prioritization: [Host {"id": "7", "name":
> > "srv-5-comp.blabla.blub",
> > >>>>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"},
> > Host
> > >>>>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
> > >>>>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
> > {"id": "6",
> > >>>>> "name": "srv-3-comp.blabla.blub", "uuid":
> > >>>>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
> > {"id": "1",
> > >>>>> "name": "srv-2-comp.blabla.blub", "uuid":
> > >>>>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> > >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for
> speed=500Mhz,
> > >>>>> Ram=1024 MB
> > >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
> > >>>>> srv-5-comp.blabla.blub, uuid: 8c753ce1-0253-49c9-9c7f-1f971a719ce6}
> > is KVM
> > >>>>> hypervisor type, no max guest limit check needed
> > >>>>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu
> capability
> > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
> > speed: 500
> > >>>>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has
> > enough
> > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> > 1073741824 ,
> > >>>>> cpuOverprovisioningFactor: 1.0
> > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
> CPU:
> > 127200
> > >>>>> and CPU after applying overprovisioning: 127200
> > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 ,
> > Requested CPU:
> > >>>>> 500
> > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
> > >>>>> 189373743104 , Requested RAM: (1.00 GB) 1073741824
> > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
> > RAM
> > >>>>> available
> > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
> from
> > host:
> > >>>>> 7, used: 90100, reserved: 0, actual total: 127200, total with
> > >>>>> overprovisioning: 127200; requested
> > cpu:500,alloc_from_last_host?:false
> > >>>>> ,considerReservedCapacity?: true
> > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
> from
> > host:
> > >>>>> 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total:
> > (250.27 GB)
> > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
> > alloc_from_last_host?:
> > >>>>> false , considerReservedCapacity?: true
> > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> > adding to
> > >>>>> list: 7
> > >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
> > >>>>> srv-4-comp.blabla.blub, uuid: 134c625d-15ab-461a-a38d-36fb0b979b5d}
> > is KVM
> > >>>>> hypervisor type, no max guest limit check needed
> > >>>>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu
> capability
> > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
> > speed: 500
> > >>>>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has
> > enough
> > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> > 1073741824 ,
> > >>>>> cpuOverprovisioningFactor: 1.0
> > >>>>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
> CPU:
> > 127200
> > >>>>> and CPU after applying overprovisioning: 127200
> > >>>>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 ,
> > Requested
> > >>>>> CPU: 500
> > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
> > >>>>> 251009040384 , Requested RAM: (1.00 GB) 1073741824
> > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
> > RAM
> > >>>>> available
> > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
> from
> > host:
> > >>>>> 5, used: 11100, reserved: 0, actual total: 127200, total with
> > >>>>> overprovisioning: 127200; requested
> > cpu:500,alloc_from_last_host?:false
> > >>>>> ,considerReservedCapacity?: true
> > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
> from
> > host:
> > >>>>> 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total:
> > (250.27 GB)
> > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
> > alloc_from_last_host?:
> > >>>>> false , considerReservedCapacity?: true
> > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> > adding to
> > >>>>> list: 5
> > >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
> > >>>>> srv-3-comp.blabla.blub, uuid: 6858713b-7e17-46d8-854d-24a13f9c7001}
> > is KVM
> > >>>>> hypervisor type, no max guest limit check needed
> > >>>>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu
> capability
> > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
> > speed: 500
> > >>>>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has
> > enough
> > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> > 1073741824 ,
> > >>>>> cpuOverprovisioningFactor: 1.0
> > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
> CPU:
> > 127200
> > >>>>> and CPU after applying overprovisioning: 127200
> > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 ,
> > Requested CPU:
> > >>>>> 500
> > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
> > >>>>> 204032835584 , Requested RAM: (1.00 GB) 1073741824
> > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
> > RAM
> > >>>>> available
> > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
> from
> > host:
> > >>>>> 6, used: 42900, reserved: 0, actual total: 127200, total with
> > >>>>> overprovisioning: 127200; requested
> > cpu:500,alloc_from_last_host?:false
> > >>>>> ,considerReservedCapacity?: true
> > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
> from
> > host:
> > >>>>> 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total:
> > (250.27 GB)
> > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
> > alloc_from_last_host?:
> > >>>>> false , considerReservedCapacity?: true
> > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> > adding to
> > >>>>> list: 6
> > >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
> > >>>>> srv-2-comp.blabla.blub, uuid: 883f7f62-55f2-42f4-8242-16b6847c4204}
> > is KVM
> > >>>>> hypervisor type, no max guest limit check needed
> > >>>>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu
> capability
> > >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
> > speed: 500
> > >>>>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has
> > enough
> > >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> > 1073741824 ,
> > >>>>> cpuOverprovisioningFactor: 1.0
> > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total
> CPU:
> > 127200
> > >>>>> and CPU after applying overprovisioning: 127200
> > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 ,
> > Requested CPU:
> > >>>>> 500
> > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
> > >>>>> 229802639360 , Requested RAM: (1.00 GB) 1073741824
> > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
> > RAM
> > >>>>> available
> > >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU
> from
> > host:
> > >>>>> 1, used: 48200, reserved: 0, actual total: 127200, total with
> > >>>>> overprovisioning: 127200; requested
> > cpu:500,alloc_from_last_host?:false
> > >>>>> ,considerReservedCapacity?: true
> > >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM
> from
> > host:
> > >>>>> 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total:
> > (250.27 GB)
> > >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
> > alloc_from_last_host?:
> > >>>>> false , considerReservedCapacity?: true
> > >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> > adding to
> > >>>>> list: 1
> > >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> > >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator
> returning 4
> > >>>>> suitable hosts
> > >>>>> 2022-08-29 10:51:56,260 DEBUG [c.c.c.CapacityManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> > >>>>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
> > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} state
> > >>>>> transited from [Starting] to [Stopped] with event
> [OperationFailed].
> > VM's
> > >>>>> original host: null, new host: null, host before state transition:
> > null
> > >>>>> 2022-08-29 10:51:56,260 WARN [c.c.v.VirtualMachineManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> > >>>>> (logid:ba10fec5) Unable to orchestrate start VM instance {id: "83",
> > name:
> > >>>>> "v-83-VM", uuid: "b1d1f9c6-eafe-4527-9335-e649c646aab0",
> > >>>>> type="ConsoleProxy"} due to [Unable to create deployment, no usable
> > volumes
> > >>>>> found for the VM: 83].
> > >>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to create
> > >>>>> deployment, no usable volumes found for the VM: 83
> > >>>>> at
> > >>>>>
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
> > >>>>> at
> > >>>>>
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
> > >>>>> at
> > >>>>>
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
> > >>>>> at
> > >>>>>
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
> > >>>>> at
> > >>>>>
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
> > >>>>> at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> > >>>>> Source)
> > >>>>> at
> > >>>>>
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > >>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > >>>>> at
> > >>>>>
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > >>>>> at
> > >>>>>
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > >>>>> at
> > >>>>>
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > >>>>> at
> > >>>>>
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > >>>>> at
> > >>>>>
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > >>>>> at
> > >>>>>
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > >>>>> at
> > >>>>>
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > >>>>> at
> > >>>>>
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > >>>>> at
> > >>>>>
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > >>>>> at
> > >>>>>
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > >>>>> at
> > >>>>>
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > >>>>> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > >>>>> at
> > >>>>>
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > >>>>> at
> > >>>>>
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > >>>>> at java.base/java.lang.Thread.run(Thread.java:829)
> > >>>>> 2022-08-29 10:51:56,262 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> > >>>>> (logid:ba10fec5) Done executing VM work job:
> > >>>>>
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
> > >>>>> 2022-08-29 10:51:56,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> > >>>>> (logid:ba10fec5) Complete async job-3539, jobStatus: FAILED,
> > resultCode: 0,
> > >>>>> result:
> > >>>>>
> >
> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODMiLCBuYW1lOiAidi04My1WTSIsIHV1aWQ6ICJiMWQxZjljNi1lYWZlLTQ1MjctOTMzNS1lNjQ5YzY0NmFhYjAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODNdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
> > >>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> > >>>>> (logid:ba10fec5) Publish async job-3539 complete on message bus
> > >>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> > >>>>> (logid:ba10fec5) Wake up jobs related to job-3539
> > >>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> > >>>>> (logid:ba10fec5) Update db status for job-3539
> > >>>>> 2022-08-29 10:51:56,266 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> > >>>>> (logid:ba10fec5) Wake up jobs joined with job-3539 and disjoin all
> > subjobs
> > >>>>> created from job- 3539
> > >>>>> 2022-08-29 10:51:56,276 DEBUG [c.c.v.VmWorkJobDispatcher]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
> > (logid:ba10fec5) Done
> > >>>>> with run of VM work job: com.cloud.vm.VmWorkStart for VM 83, job
> > origin:
> > >>>>> 3445
> > >>>>> 2022-08-29 10:51:56,276 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
> > (logid:ba10fec5) Done
> > >>>>> executing com.cloud.vm.VmWorkStart for job-3539
> > >>>>> 2022-08-29 10:51:56,278 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
> > (logid:ba10fec5)
> > >>>>> Remove job-3539 from job monitoring
> > >>>>> 2022-08-29 10:51:56,285 WARN [c.c.c.ConsoleProxyManagerImpl]
> > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Unable to allocate
> > console
> > >>>>> proxy standby capacity for zone [1] due to [Unable to orchestrate
> > start VM
> > >>>>> instance {id: "83", name: "v-83-VM", uuid:
> > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to
> > [Unable
> > >>>>> to create deployment, no usable volumes found for the VM: 83].].
> > >>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to
> > orchestrate
> > >>>>> start VM instance {id: "83", name: "v-83-VM", uuid:
> > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to
> > [Unable
> > >>>>> to create deployment, no usable volumes found for the VM: 83].
> > >>>>> at
> > >>>>>
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> > >>>>> at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> > >>>>> Source)
> > >>>>> at
> > >>>>>
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > >>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > >>>>> at
> > >>>>>
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > >>>>> at
> > >>>>>
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > >>>>> at
> > >>>>>
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > >>>>> at
> > >>>>>
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > >>>>> at
> > >>>>>
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > >>>>> at
> > >>>>>
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > >>>>> at
> > >>>>>
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > >>>>> at
> > >>>>>
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > >>>>> at
> > >>>>>
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > >>>>> at
> > >>>>>
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > >>>>> at
> > >>>>>
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > >>>>> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > >>>>> at
> > >>>>>
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > >>>>> at
> > >>>>>
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > >>>>> at java.base/java.lang.Thread.run(Thread.java:829)
> > >>>>> 2022-08-29 10:51:56,287 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) received console
> > proxy alert
> > >>>>> 2022-08-29 10:51:56,288 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Console proxy
> creation
> > >>>>> failure, zone: blabla
> > >>>>> 2022-08-29 10:51:56,289 WARN [c.c.a.AlertManagerImpl]
> > >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) alertType=[10]
> > >>>>> dataCenterId=[1] podId=[null] clusterId=[null] message=[Console
> proxy
> > >>>>> creation failure. zone: blabla, error details: Unable to
> orchestrate
> > start
> > >>>>> VM instance {id: "83", name: "v-83-VM", uuid:
> > >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to
> > [Unable
> > >>>>> to create deployment, no usable volumes found for the VM: 83].].
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> Am Mo., 29. Aug. 2022 um 09:40 Uhr schrieb Slavka Peleva
> > >>>>> <sl...@storpool.com.invalid>:
> > >>>>>
> > >>>>>> Hi Chris,
> > >>>>>>
> > >>>>>> Did you recreate the system VMs? In the 4.17 version, the
> > >>>>>> `systemvm.iso` is
> > >>>>>> deprecated. In its place, you should have the file on agents
> > >>>>>> `/usr/share/cloudstack-common/vms/agent.zip`.
> > >>>>>> Can you share the complete log if the system VMs are newly created
> > >>>>>> after
> > >>>>>> the upgrade?
> > >>>>>>
> > >>>>>> Best regards,
> > >>>>>> Slavka
> > >>>>>>
> > >>>>>> On Sun, Aug 28, 2022 at 2:16 PM Vash_X@gmx.de <Va...@gmx.de>
> > wrote:
> > >>>>>>
> > >>>>>> > Hi everyone,
> > >>>>>> >
> > >>>>>> > faceing some challanges again after upgradeing to 4.17.
> > >>>>>> > Did as explained in the docs.
> > >>>>>> >
> > >>>>>> > Afterwards I am now not able to migrate system VMs to different
> > >>>>>> hosts.
> > >>>>>> > Getting the error
> > >>>>>> > "Exception during migrate: org.libvirt.LibvirtException: Cannot
> > >>>>>> access
> > >>>>>> > storage file '/usr/share/cloudstack-common/vms/systemvm.iso': No
> > >>>>>> such file
> > >>>>>> > or directory"
> > >>>>>> >
> > >>>>>> > Checked the exitstence of the file on the servers:
> > >>>>>> > Mangement-Server has a "systemvm.iso.bak"
> > >>>>>> > Hosts - no "systemvm.iso" at all present.
> > >>>>>> >
> > >>>>>> > Maybe someone has an idea on how to move on with this?
> > >>>>>> >
> > >>>>>> > Regards,
> > >>>>>> > Chris
> > >>>>>> >
> > >>>>>>
> > >>>>>
> >
>
Re: Error migrate System VMs after upgrageing to 4.17
Posted by Nicolas Vazquez <Ni...@shapeblue.com>.
Hi Chris,
What is the value for the setting ‘system.vm.public.ip.reservation.mode.strictness’?
Looks like the error is not being able to get an IP address for zone id=1, vlanId id=[3, 10, 13, 14].
If I understood correctly you have defined a range for system VMs? Can you please check if this query gets any result?
select i.public_ip_address, i.vlan_db_id, v.vlan_id from user_ip_address i join vlan v on (i.vlan_db_id = v.id)
where i.forsystemvms = 1 and i.allocated is null and i.state = 'Free' and v.removed is null and i.data_center_id = 1;
Regards,
Nicolas Vazquez
From: Vash_X@gmx.de <Va...@gmx.de>
Date: Tuesday, 30 August 2022 at 09:38
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>, Slavka Peleva <sl...@storpool.com>
Subject: Re: Error migrate System VMs after upgrageing to 4.17
Hi Slavka,
that was my first thought, too. I checked beforehand and confirmed with
your provided SQL Statements:
*`select * from user_ip_address where forsystemvms = 1;`* <---- returns my
defined (Public) IPs out of the correct range for SystemVMs
*`SELECT * FROM clou*d.user_ip_address where allocated IS NULL AND removed
IS NULL; <-- All IPs shown beside the one which is currently in use for the
running ssvm. Meaning from my point of view "free" for allocation for other
system VMs.
Querys performed directly on the DB-Server.
I can't see a reason why there shouldn't be any IPs left for allocation.
Also it is interesting why the "reasoning" for the failing deployment
changes from "no IP availeable" to "no useable volumes found".
How ever i can't see anything missing at the moment, what makes me even
more puzzeled.
Regards
Chris
Am Di., 30. Aug. 2022 um 13:31 Uhr schrieb Slavka Peleva
<sl...@storpool.com.invalid>:
> Hi Chris,
>
> The deployment fails because CS cannot find a free IP address to allocate
> for the system VM. Can you check if you explicitly added a range for the
> system VMs, or you can check with this `select * from user_ip_address where
> forsystemvms = 1;` If you don't have any ranges, you can get free IPs
> with `SELECT
> * FROM cloud.user_ip_address where allocated IS NULL AND removed IS NULL;`
> . To get the exact query executed by CS, you need to increase the log level
> to TRACE.
>
> Best regards,
> Slavka
>
> On Mon, Aug 29, 2022 at 8:03 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
>
> > Hi Slavka,
> >
> > no, i am not using any kind of custome offerings.
> >
> > I try to provide a more detailled log this time.
> > Situation:
> > v-83-VM was the "broken" consoleproxy. So the following log starts with
> > it's expunging.
> > Afterwards the new consoleproxy shall be deployed - v-85-VM.
> >
> > This first deployment (logid:dcfe0999) fails due to Insufficent address
> > capacity - which i can't really understand or see in the database,
> checking
> > the availeable IPs for systemVMs (given that i might looking up the wrong
> > tables....).
> > How ever CS is working as expected trying to deploy the console proxy
> > (logid:440fd101) Sync job-4288).
> > That job fails with "Unable to create deployment, no usable volumes found
> > for the VM: 85".
> > Afterwards i have another try to start the consoleproxy (job-4289) which
> > fails for the same reason. And thats how things go on in the log file...
> >
> >
> > 2022-08-29 17:01:16,224 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> > Cleaning up hypervisor data structures (ex. SRs in XenServer) for managed
> > storage. Data from VM instance {id: "83", name: "v-83-VM", uuid:
> > "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}.
> > 2022-08-29 17:01:16,225 DEBUG [o.a.c.e.o.VolumeOrchestrator]
> > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> > Cleaning storage for vm: 83
> > 2022-08-29 17:01:16,233 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> > Expunged VM instance {id: "83", name: "v-83-VM", uuid:
> > "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
> > 2022-08-29 17:01:16,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> > Complete async job-4286, jobStatus: SUCCEEDED, resultCode: 0, result:
> >
> org.apache.cloudstack.api.response.SystemVmResponse/systemvm/{"id":"b1d1f9c6-eafe-4527-9335-e649c646aab0","systemvmtype":"consoleproxy","zoneid":"48e2e928-3300-43b5-8e3a-d9b358022502","zonename":"xxxxxxxxxx","dns1":"10.129.0.4","dns2":"195.10.195.195","name":"v-83-VM","templateid":"c20799e4-a990-449a-aae6-4b67374c7aff","templatename":"systemvm-kvm-4.17.0","created":"2022-08-29T10:38:26+0200","state":"Stopped","isdynamicallyscalable":"false","hasannotations":"false"}
> > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> > Publish async job-4286 complete on message bus
> > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> > Wake up jobs related to job-4286
> > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> > Update db status for job-4286
> > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> > Wake up jobs joined with job-4286 and disjoin all subjobs created from
> job-
> > 4286
> > 2022-08-29 17:01:16,260 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-35:ctx-29e06f28 job-4286) (logid:5fe8559a) Done
> executing
> > org.apache.cloudstack.api.command.admin.systemvm.DestroySystemVmCmd for
> > job-4286
> > 2022-08-29 17:01:16,260 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > (API-Job-Executor-35:ctx-29e06f28 job-4286) (logid:5fe8559a) Remove
> > job-4286 from job monitoring
> > 2022-08-29 17:01:17,691 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-6:ctx-6af18142) (logid:fe263a70) HA health
> check
> > task is running...
> >
> > 2022-08-29 17:01:21,694 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-6:ctx-89b1442f) (logid:0efc40b2) HA health
> check
> > task is running...
> > 2022-08-29 17:01:25,698 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-5:ctx-62313d05) (logid:7073663f) HA health
> check
> > task is running...
> > 2022-08-29 17:01:25,767 DEBUG [c.c.s.StatsCollector]
> > (secstorage-1:ctx-c80e5a6e) (logid:f8e3eb60) Verifying image storage [1].
> > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
> > 2022-08-29 17:01:25,773 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > (secstorage-1:ctx-c80e5a6e) (logid:f8e3eb60) Zone [1] is ready to launch
> > secondary storage VM.
> > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Zone 1 is ready to launch
> > console proxy
> > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Expand console proxy
> standby
> > capacity for zone xxxxxxxxxx
> > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating console proxy
> > standby capacity for zone [1].
> > 2022-08-29 17:01:26,032 INFO [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) No stopped console proxy
> is
> > available, need to allocate a new console proxy
> > 2022-08-29 17:01:26,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Assign console proxy from
> a
> > newly started instance for request from data center : 1
> > 2022-08-29 17:01:26,056 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing network
> > configuration for offering [Network Offering
> > [1-Public-System-Public-Network]: Network {"id": 200, "name": "null",
> > "uuid": "1b0bf965-a391-478d-833d-3e37e4a15eb8", "networkofferingid": 1}
> > 2022-08-29 17:01:26,056 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
> > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account {"id": 1,
> > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
> > 2022-08-29 17:01:26,064 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing network
> > configuration for offering [Network Offering
> > [3-Control-System-Control-Network]: Network {"id": 202, "name": "null",
> > "uuid": "2af42518-fbf4-4a35-9d6b-370e6ad315c7", "networkofferingid": 3}
> > 2022-08-29 17:01:26,064 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
> > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account {"id": 1,
> > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
> > 2022-08-29 17:01:26,069 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing network
> > configuration for offering [Network Offering
> > [2-Management-System-Management-Network]: Network {"id": 201, "name":
> > "null", "uuid": "53711d87-a3c6-4f33-bfce-c30e0de72760",
> > "networkofferingid": 2}
> > 2022-08-29 17:01:26,069 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
> > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account {"id": 1,
> > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
> > 2022-08-29 17:01:26,077 INFO [c.c.v.VirtualMachineManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) allocating virtual machine
> > from template:c20799e4-a990-449a-aae6-4b67374c7aff with hostname:v-85-VM
> > and 3 networks
> > 2022-08-29 17:01:26,079 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating entries for VM:
> > VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
> > 2022-08-29 17:01:26,082 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating nics for VM
> > instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
> > 2022-08-29 17:01:26,083 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating nic for vm VM
> > instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} in network
> > Network {"id": 200, "name": "null", "uuid":
> > "1b0bf965-a391-478d-833d-3e37e4a15eb8", "networkofferingid": 1} with
> > requested profile NicProfile
> >
> {"broadcastUri":null,"iPv4Address":null,"id":0,"reservationId":null,"vmId":0}
> > 2022-08-29 17:01:26,089 WARN [c.c.n.IpAddressManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to get ip address
> in
> > zone id=1, vlanId id=[3, 10, 13, 14]
> > 2022-08-29 17:01:26,090 DEBUG [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Rolling back the
> > transaction: Time = 8 Name = consoleproxy-1; called by
> >
> -TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-Transaction.execute:38-IpAddressManagerImpl.listAvailablePublicIps:824-IpAddressManagerImpl.fetchNewPublicIp:811-IpAddressManagerImpl.fetchNewPublicIp:803-IpAddressManagerImpl.assignPublicIpAddress:784-PublicNetworkGuru.getIp:125-PublicNetworkGuru.allocate:172-NetworkOrchestrator.allocateNic:973-NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice:823
> > 2022-08-29 17:01:26,090 WARN [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to allocate proxy
> > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
> > due to [Insufficient address capacity].
> > com.cloud.exception.InsufficientAddressCapacityException: Insufficient
> > address capacityScope=interface com.cloud.dc.DataCenter; id=1
> > at
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
> > at
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
> > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> > at
> >
> com.cloud.network.IpAddressManagerImpl.listAvailablePublicIps(IpAddressManagerImpl.java:824)
> > at
> >
> com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl.java:811)
> > at
> >
> com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl.java:803)
> > at
> >
> com.cloud.network.IpAddressManagerImpl.assignPublicIpAddress(IpAddressManagerImpl.java:784)
> > at
> >
> com.cloud.network.guru.PublicNetworkGuru.getIp(PublicNetworkGuru.java:125)
> > at
> >
> com.cloud.network.guru.PublicNetworkGuru.allocate(PublicNetworkGuru.java:172)
> > at
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocateNic(NetworkOrchestrator.java:973)
> > at
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice(NetworkOrchestrator.java:823)
> > at
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$3.doInTransactionWithoutResult(NetworkOrchestrator.java:797)
> > at
> >
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
> > at
> >
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
> > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> > at
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocate(NetworkOrchestrator.java:778)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl$1.doInTransactionWithoutResult(VirtualMachineManagerImpl.java:478)
> > at
> >
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
> > at
> >
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
> > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:469)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:528)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:700)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
> > at
> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
> > at
> > com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
> > at
> >
> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > at
> >
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> > at
> >
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > at java.base/java.lang.Thread.run(Thread.java:829)
> > 2022-08-29 17:01:26,091 WARN [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to allocate console
> > proxy standby capacity for zone [1] due to [Unable to allocate proxy
> > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
> > due to [Insufficient address capacity].].
> > com.cloud.utils.exception.CloudRuntimeException: Unable to allocate proxy
> > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
> > due to [Insufficient address capacity].
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:704)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
> > at
> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
> > at
> > com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
> > at
> >
> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > at
> >
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> > at
> >
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > at java.base/java.lang.Thread.run(Thread.java:829)
> > Caused by: com.cloud.exception.InsufficientAddressCapacityException:
> > Insufficient address capacityScope=interface com.cloud.dc.DataCenter;
> id=1
> > at
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
> > at
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
> > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> > ... 38 more
> > 2022-08-29 17:01:26,091 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) received console proxy
> alert
> > 2022-08-29 17:01:26,092 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Console proxy creation
> > failure, zone: xxxxxxxxxx
> > 2022-08-29 17:01:26,093 WARN [c.c.a.AlertManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) alertType=[10]
> > dataCenterId=[1] podId=[null] clusterId=[null] message=[Console proxy
> > creation failure. zone: xxxxxxxxxx, error details: Unable to allocate
> proxy
> > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
> > due to [Insufficient address capacity].].
> > 2022-08-29 17:01:26,096 WARN [c.c.a.AlertManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) No recipients set in
> global
> > setting 'alert.email.addresses', skipping sending alert with subject
> > [Console proxy creation failure. zone: xxxxxxxxxx, error details: Unable
> to
> > allocate proxy [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
> > due to [Insufficient address capacity].] and content [Console proxy
> > creation failure (zone xxxxxxxxxx)].
> > 2022-08-29 17:01:26,096 WARN [c.c.v.SystemVmLoadScanner]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unexpected exception
> Unable
> > to allocate proxy [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
> > due to [Insufficient address capacity].
> > com.cloud.utils.exception.CloudRuntimeException: Unable to allocate proxy
> > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
> > due to [Insufficient address capacity].
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:704)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
> > at
> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
> > at
> > com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
> > at
> >
> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > at
> >
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> > at
> >
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > at java.base/java.lang.Thread.run(Thread.java:829)
> > Caused by: com.cloud.exception.InsufficientAddressCapacityException:
> > Insufficient address capacityScope=interface com.cloud.dc.DataCenter;
> id=1
> > at
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
> > at
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
> > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> > ... 38 more
> > 2022-08-29 17:01:29,700 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-5:ctx-994465bd) (logid:1e2822fd) HA health
> check
> > task is running...
> > 2022-08-29 17:01:30,444 DEBUG
> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 2 routers to
> > update status.
> > 2022-08-29 17:01:30,449 DEBUG
> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 1 VPC's to
> > update Redundant State.
> > 2022-08-29 17:01:30,451 DEBUG
> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 0 networks to
> > update RvR status.
> > 2022-08-29 17:01:30,459 DEBUG [c.c.a.t.Request]
> > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
> > 6-4833769775052102463: Sending { Cmd , MgmtId: 90520740759984, via:
> > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
> router.name
> ":"r-79-VM","router.ip":"169.254.144.198"},"wait":"30","bypassHostMaintenance":"false"}}]
> > }
> > 2022-08-29 17:01:30,643 DEBUG [c.c.a.t.Request]
> > (AgentManager-Handler-12:null) (logid:) Seq 6-4833769775052102463:
> > Processing: { Ans: , MgmtId: 90520740759984, via: 6, Ver: v1, Flags: 10,
> >
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"PRIMARY","result":"true","details":"Status:
> > PRIMARY
> > ","wait":"0","bypassHostMaintenance":"false"}}] }
> > 2022-08-29 17:01:30,643 DEBUG [c.c.a.t.Request]
> > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
> > 6-4833769775052102463: Received: { Ans: , MgmtId: 90520740759984, via:
> > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer } }
> > 2022-08-29 17:01:30,643 DEBUG [c.c.a.m.AgentManagerImpl]
> > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Details
> from
> > executing class com.cloud.agent.api.CheckRouterCommand: Status: PRIMARY
> >
> > 2022-08-29 17:01:30,655 DEBUG [c.c.a.t.Request]
> > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
> > 1-5410793477308947401: Sending { Cmd , MgmtId: 90520740759984, via:
> > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
> router.name
> ":"r-80-VM","router.ip":"169.254.217.200"},"wait":"30","bypassHostMaintenance":"false"}}]
> > }
> > 2022-08-29 17:01:30,796 DEBUG [c.c.a.t.Request]
> > (AgentManager-Handler-7:null) (logid:) Seq 1-5410793477308947401:
> > Processing: { Ans: , MgmtId: 90520740759984, via: 1, Ver: v1, Flags: 10,
> >
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"BACKUP","result":"true","details":"Status:
> > BACKUP
> > ","wait":"0","bypassHostMaintenance":"false"}}] }
> > 2022-08-29 17:01:30,796 DEBUG [c.c.a.t.Request]
> > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
> > 1-5410793477308947401: Received: { Ans: , MgmtId: 90520740759984, via:
> > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer } }
> > 2022-08-29 17:01:30,796 DEBUG [c.c.a.m.AgentManagerImpl]
> > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Details
> from
> > executing class com.cloud.agent.api.CheckRouterCommand: Status: BACKUP
> >
> > 2022-08-29 17:01:33,703 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-5:ctx-70bca202) (logid:b332dbe0) HA health
> check
> > task is running...
> > 2022-08-29 17:01:34,083 DEBUG [c.c.s.StatsCollector]
> > (StatsCollector-6:ctx-213e2ecc) (logid:aa0584c9) AutoScaling Monitor is
> > running...
> > 2022-08-29 17:01:37,707 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-6:ctx-57493517) (logid:5093e75f) HA health
> check
> > task is running...
> > 2022-08-29 17:01:38,640 DEBUG [c.c.s.StatsCollector]
> > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) HostStatsCollector is
> > running...
> > 2022-08-29 17:01:38,655 DEBUG [c.c.a.t.Request]
> > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
> 1-5410793477308947402:
> > Received: { Ans: , MgmtId: 90520740759984, via:
> > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> > 2022-08-29 17:01:38,707 DEBUG [c.c.a.t.Request]
> > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
> 5-8959067033723473395:
> > Received: { Ans: , MgmtId: 90520740759984, via:
> > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> > 2022-08-29 17:01:38,717 DEBUG [c.c.a.t.Request]
> > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
> 6-4833769775052102464:
> > Received: { Ans: , MgmtId: 90520740759984, via:
> > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> > 2022-08-29 17:01:38,725 DEBUG [c.c.a.t.Request]
> > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
> 7-295548725546192472:
> > Received: { Ans: , MgmtId: 90520740759984, via:
> > 7(srv-5-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> > 2022-08-29 17:01:41,709 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-5:ctx-a65ae462) (logid:bc2b7320) HA health
> check
> > task is running...
> > 2022-08-29 17:01:45,712 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-2:ctx-05491ace) (logid:b2232c4d) HA health
> check
> > task is running...
> > 2022-08-29 17:01:47,019 DEBUG [c.c.a.m.AgentManagerImpl]
> > (AgentManager-Handler-14:null) (logid:) Ping from Routing host
> > 7(srv-5-comp.aaaaaaaa.bbb)
> >
> > 2022-08-29 17:01:49,715 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-2:ctx-7943581c) (logid:504c98ff) HA health
> check
> > task is running...
> > 2022-08-29 17:01:51,881 DEBUG [c.c.a.m.AgentManagerImpl]
> > (AgentManager-Handler-13:null) (logid:) Ping from Routing host
> > 1(srv-2-comp.aaaaaaaa.bbb)
> >
> > 2022-08-29 17:01:53,718 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-4:ctx-b11cfa22) (logid:3163613f) HA health
> check
> > task is running...
> > 2022-08-29 17:01:53,955 DEBUG [c.c.a.t.Request]
> > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
> 1-5410793477308947403:
> > Received: { Ans: , MgmtId: 90520740759984, via:
> > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
> > 2022-08-29 17:01:54,025 DEBUG [c.c.a.t.Request]
> > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
> 5-8959067033723473396:
> > Received: { Ans: , MgmtId: 90520740759984, via:
> > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
> > 2022-08-29 17:01:54,096 DEBUG [c.c.a.t.Request]
> > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
> 6-4833769775052102465:
> > Received: { Ans: , MgmtId: 90520740759984, via:
> > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
> > 2022-08-29 17:01:54,217 DEBUG [c.c.a.t.Request]
> > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
> 7-295548725546192473:
> > Received: { Ans: , MgmtId: 90520740759984, via:
> > 7(srv-5-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
> > 2022-08-29 17:01:55,765 DEBUG [c.c.s.StatsCollector]
> > (secstorage-1:ctx-7d9995d7) (logid:e388a8b8) Verifying image storage [1].
> > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
> > 2022-08-29 17:01:55,771 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > (secstorage-1:ctx-7d9995d7) (logid:e388a8b8) Zone [1] is ready to launch
> > secondary storage VM.
> > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Zone 1 is ready to launch
> > console proxy
> > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Expand console proxy
> standby
> > capacity for zone xxxxxxxxxx
> > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Allocating console proxy
> > standby capacity for zone [1].
> > 2022-08-29 17:01:56,034 INFO [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Found a stopped console
> > proxy, starting it. Vm id : 85
> > 2022-08-29 17:01:56,035 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) start parameter value of
> > enterHardwareSetup == <very null> during processing of queued job
> > 2022-08-29 17:01:56,044 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Sync job-4288 execution on
> > object VmWorkJobQueue.85
> > 2022-08-29 17:01:56,173 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-e75826b1) (logid:e80fe659) Execute
> sync-queue
> > item: SyncQueueItemVO {id:2105, queueId: 2105, contentType: AsyncJob,
> > contentId: 4288, lastProcessMsid: 90520740759984, lastprocessNumber: 1,
> > lastProcessTime: Mon Aug 29 17:01:56 CEST 2022, created: Mon Aug 29
> > 17:01:56 CEST 2022}
> > 2022-08-29 17:01:56,174 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-e75826b1) (logid:e80fe659) Schedule queued
> > job-4288
> > 2022-08-29 17:01:56,179 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:7e671dd0)
> Add
> > job-4288 into job monitoring
> > 2022-08-29 17:01:56,186 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5)
> > Executing AsyncJobVO: {id:4288, userId: 1, accountId: 1, instanceType:
> > null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
> >
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAVXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
> > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> > result: null, initMsid: 90520740759984, completeMsid: null, lastUpdated:
> > null, lastPolled: null, created: Mon Aug 29 17:01:56 CEST 2022, removed:
> > null}
> > 2022-08-29 17:01:56,186 DEBUG [c.c.v.VmWorkJobDispatcher]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5)
> Run
> > VM work job: com.cloud.vm.VmWorkStart for VM 85, job origin: 3445
> > 2022-08-29 17:01:56,187 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Execute VM work job:
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> > 2022-08-29 17:01:56,189 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) orchestrating VM start for 'v-85-VM'
> > com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
> > 2022-08-29 17:01:56,196 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
> > transited from [Stopped] to [Starting] with event [StartRequested]. VM's
> > original host: null, new host: null, host before state transition: null
> > 2022-08-29 17:01:56,196 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Successfully transitioned to start state for VM instance
> > {id: "85", name: "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
> > type="ConsoleProxy"} reservation id =
> 0ab68202-d8be-469c-9b2a-87a55c2dc489
> > 2022-08-29 17:01:56,200 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts: null
> > 2022-08-29 17:01:56,202 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) VM start attempt #1
> > 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
> > 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Trying to allocate a host and storage pools from dc:1,
> > pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
> > 1073741824
> > 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
> > 2022-08-29 17:01:56,212 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
> > 2022-08-29 17:01:56,214 DEBUG [c.c.d.FirstFitPlanner]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Searching all possible resources under this Zone: 1
> > 2022-08-29 17:01:56,215 DEBUG [c.c.d.FirstFitPlanner]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Listing clusters in order of aggregate capacity, that
> have
> > (at least one host with) enough CPU and RAM capacity under this Zone: 1
> > 2022-08-29 17:01:56,218 DEBUG [c.c.d.FirstFitPlanner]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Removing from the clusterId list these clusters from
> avoid
> > set: []
> > 2022-08-29 17:01:56,224 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
> > 2022-08-29 17:01:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested with
> > Custom[UEFI] Boot Type false
> > 2022-08-29 17:01:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in dc: 1
> > pod:1 cluster:1
> > 2022-08-29 17:01:56,226 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4 hosts
> to
> > check for allocation: [Host {"id": "5", "name":
> "srv-4-comp.aaaaaaaa.bbb",
> > "uuid": "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
> > {"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb", "uuid":
> > "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host {"id":
> "6",
> > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id":
> "1",
> > "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> > 2022-08-29 17:01:56,232 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for allocation
> > after prioritization: [Host {"id": "5", "name":
> "srv-4-comp.aaaaaaaa.bbb",
> > "uuid": "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
> > {"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb", "uuid":
> > "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host {"id":
> "6",
> > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id":
> "1",
> > "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> > 2022-08-29 17:01:56,232 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for speed=500Mhz,
> > Ram=1024 MB
> > 2022-08-29 17:01:56,232 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
> > srv-4-comp.aaaaaaaa.bbb, uuid: 134c625d-15ab-461a-a38d-36fb0b979b5d} is
> KVM
> > hypervisor type, no max guest limit check needed
> > 2022-08-29 17:01:56,234 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu capability
> > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> > 2022-08-29 17:01:56,234 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has enough
> > capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> > cpuOverprovisioningFactor: 1.0
> > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> > and CPU after applying overprovisioning: 127200
> > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 , Requested
> > CPU: 500
> > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
> > 251009040384 , Requested RAM: (1.00 GB) 1073741824
> > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > available
> > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> > 5, used: 11100, reserved: 0, actual total: 127200, total with
> > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > ,considerReservedCapacity?: true
> > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> > 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total: (250.27
> GB)
> > 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> > false , considerReservedCapacity?: true
> > 2022-08-29 17:01:56,235 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding
> to
> > list: 5
> > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
> > srv-5-comp.aaaaaaaa.bbb, uuid: 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is
> KVM
> > hypervisor type, no max guest limit check needed
> > 2022-08-29 17:01:56,238 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu capability
> > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> > 2022-08-29 17:01:56,238 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has enough
> > capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> > cpuOverprovisioningFactor: 1.0
> > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> > and CPU after applying overprovisioning: 127200
> > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 , Requested
> CPU:
> > 500
> > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
> > 189373743104 , Requested RAM: (1.00 GB) 1073741824
> > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > available
> > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> > 7, used: 90100, reserved: 0, actual total: 127200, total with
> > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > ,considerReservedCapacity?: true
> > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> > 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total: (250.27
> GB)
> > 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> > false , considerReservedCapacity?: true
> > 2022-08-29 17:01:56,240 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding
> to
> > list: 7
> > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
> > srv-3-comp.aaaaaaaa.bbb, uuid: 6858713b-7e17-46d8-854d-24a13f9c7001} is
> KVM
> > hypervisor type, no max guest limit check needed
> > 2022-08-29 17:01:56,242 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu capability
> > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> > 2022-08-29 17:01:56,242 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has enough
> > capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> > cpuOverprovisioningFactor: 1.0
> > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> > and CPU after applying overprovisioning: 127200
> > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 , Requested
> CPU:
> > 500
> > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
> > 204032835584 , Requested RAM: (1.00 GB) 1073741824
> > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > available
> > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> > 6, used: 42900, reserved: 0, actual total: 127200, total with
> > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > ,considerReservedCapacity?: true
> > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> > 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total: (250.27
> GB)
> > 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> > false , considerReservedCapacity?: true
> > 2022-08-29 17:01:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding
> to
> > list: 6
> > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
> > srv-2-comp.aaaaaaaa.bbb, uuid: 883f7f62-55f2-42f4-8242-16b6847c4204} is
> KVM
> > hypervisor type, no max guest limit check needed
> > 2022-08-29 17:01:56,246 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu capability
> > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> > 2022-08-29 17:01:56,246 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has enough
> > capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> > cpuOverprovisioningFactor: 1.0
> > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> > and CPU after applying overprovisioning: 127200
> > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 , Requested
> CPU:
> > 500
> > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
> > 229802639360 , Requested RAM: (1.00 GB) 1073741824
> > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > available
> > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> > 1, used: 48200, reserved: 0, actual total: 127200, total with
> > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > ,considerReservedCapacity?: true
> > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> > 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total: (250.27
> GB)
> > 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> > false , considerReservedCapacity?: true
> > 2022-08-29 17:01:56,247 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding
> to
> > list: 1
> > 2022-08-29 17:01:56,247 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator returning 4
> > suitable hosts
> > 2022-08-29 17:01:56,258 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
> > transited from [Starting] to [Stopped] with event [OperationFailed]. VM's
> > original host: null, new host: null, host before state transition: null
> > 2022-08-29 17:01:56,258 WARN [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Unable to orchestrate start VM instance {id: "85", name:
> > "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
> > type="ConsoleProxy"} due to [Unable to create deployment, no usable
> volumes
> > found for the VM: 85].
> > com.cloud.utils.exception.CloudRuntimeException: Unable to create
> > deployment, no usable volumes found for the VM: 85
> > at
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
> > at
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
> > at
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
> > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
> > at
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > at java.base/java.lang.Thread.run(Thread.java:829)
> > 2022-08-29 17:01:56,259 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Done executing VM work job:
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> > 2022-08-29 17:01:56,260 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Complete async job-4288, jobStatus: FAILED, resultCode:
> 0,
> > result:
> >
> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODUiLCBuYW1lOiAidi04NS1WTSIsIHV1aWQ6ICIxMTBkODYyMC1jMDI5LTRjYTktOTA4NC0yMTAwOTk3MjVhODAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODVdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
> > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Publish async job-4288 complete on message bus
> > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Wake up jobs related to job-4288
> > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Update db status for job-4288
> > 2022-08-29 17:01:56,262 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Wake up jobs joined with job-4288 and disjoin all
> subjobs
> > created from job- 4288
> > 2022-08-29 17:01:56,270 DEBUG [c.c.v.VmWorkJobDispatcher]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5)
> > Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 85, job
> > origin: 3445
> > 2022-08-29 17:01:56,270 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5)
> > Done executing com.cloud.vm.VmWorkStart for job-4288
> > 2022-08-29 17:01:56,272 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5)
> > Remove job-4288 from job monitoring
> > 2022-08-29 17:01:56,277 WARN [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Unable to allocate console
> > proxy standby capacity for zone [1] due to [Unable to orchestrate start
> VM
> > instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> [Unable
> > to create deployment, no usable volumes found for the VM: 85].].
> > com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate
> > start VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> [Unable
> > to create deployment, no usable volumes found for the VM: 85].
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
> > at
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > at java.base/java.lang.Thread.run(Thread.java:829)
> > 2022-08-29 17:01:56,277 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) received console proxy
> alert
> > 2022-08-29 17:01:56,278 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Console proxy creation
> > failure, zone: xxxxxxxxxx
> > 2022-08-29 17:01:56,279 WARN [c.c.a.AlertManagerImpl]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) alertType=[10]
> > dataCenterId=[1] podId=[null] clusterId=[null] message=[Console proxy
> > creation failure. zone: xxxxxxxxxx, error details: Unable to orchestrate
> > start VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> [Unable
> > to create deployment, no usable volumes found for the VM: 85].].
> > 2022-08-29 17:01:56,284 WARN [c.c.a.AlertManagerImpl]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) No recipients set in
> global
> > setting 'alert.email.addresses', skipping sending alert with subject
> > [Console proxy creation failure. zone: xxxxxxxxxx, error details: Unable
> to
> > orchestrate start VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> [Unable
> > to create deployment, no usable volumes found for the VM: 85].] and
> content
> > [Console proxy creation failure (zone xxxxxxxxxx)].
> > 2022-08-29 17:01:56,284 WARN [c.c.v.SystemVmLoadScanner]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Unexpected exception
> Unable
> > to orchestrate start VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> [Unable
> > to create deployment, no usable volumes found for the VM: 85].
> > com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate
> > start VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> [Unable
> > to create deployment, no usable volumes found for the VM: 85].
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
> > at
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > at java.base/java.lang.Thread.run(Thread.java:829)
> > 2022-08-29 17:01:57,721 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-4:ctx-9528226f) (logid:4512ff9b) HA health
> check
> > task is running...
> > 2022-08-29 17:01:59,504 DEBUG [c.c.s.StatsCollector]
> > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) StorageCollector is
> > running...
> > 2022-08-29 17:01:59,518 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
> getCommandHostDelegation:
> > class com.cloud.agent.api.GetStorageStatsCommand
> > 2022-08-29 17:01:59,518 DEBUG [c.c.h.XenServerGuru]
> > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning the
> > default host to execute commands because the command is not of Copy type.
> > 2022-08-29 17:01:59,571 DEBUG [c.c.a.t.Request]
> > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
> > 11-7523826127475836583: Received: { Ans: , MgmtId: 90520740759984, via:
> > 11(s-32-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> > 2022-08-29 17:01:59,574 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
> getCommandHostDelegation:
> > class com.cloud.agent.api.GetStorageStatsCommand
> > 2022-08-29 17:01:59,574 DEBUG [c.c.h.XenServerGuru]
> > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning the
> > default host to execute commands because the command is not of Copy type.
> > 2022-08-29 17:01:59,615 DEBUG [c.c.a.t.Request]
> > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
> 5-8959067033723473397:
> > Received: { Ans: , MgmtId: 90520740759984, via:
> > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetStorageStatsAnswer
> } }
> > 2022-08-29 17:01:59,624 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
> getCommandHostDelegation:
> > class com.cloud.agent.api.GetStorageStatsCommand
> > 2022-08-29 17:01:59,624 DEBUG [c.c.h.XenServerGuru]
> > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning the
> > default host to execute commands because the command is not of Copy type.
> > 2022-08-29 17:01:59,653 DEBUG [c.c.a.t.Request]
> > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
> 6-4833769775052102466:
> > Received: { Ans: , MgmtId: 90520740759984, via:
> > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetStorageStatsAnswer
> } }
> > 2022-08-29 17:01:59,979 DEBUG [c.c.a.m.AgentManagerImpl]
> > (AgentManager-Handler-12:null) (logid:) Ping from Routing host
> > 5(srv-4-comp.aaaaaaaa.bbb)
> >
> > 2022-08-29 17:02:00,269 DEBUG [c.c.s.StatsCollector]
> > (StatsCollector-5:ctx-d2b6ffd3) (logid:95201678) DbCollector is
> running...
> > 2022-08-29 17:02:00,443 DEBUG
> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 2 routers to
> > update status.
> > 2022-08-29 17:02:00,449 DEBUG
> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 1 VPC's to
> > update Redundant State.
> > 2022-08-29 17:02:00,450 DEBUG
> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 0 networks to
> > update RvR status.
> > 2022-08-29 17:02:00,457 DEBUG [c.c.a.t.Request]
> > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
> > 6-4833769775052102467: Sending { Cmd , MgmtId: 90520740759984, via:
> > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
> router.name
> ":"r-79-VM","router.ip":"169.254.144.198"},"wait":"30","bypassHostMaintenance":"false"}}]
> > }
> > 2022-08-29 17:02:00,599 DEBUG [c.c.a.t.Request]
> > (AgentManager-Handler-7:null) (logid:) Seq 6-4833769775052102467:
> > Processing: { Ans: , MgmtId: 90520740759984, via: 6, Ver: v1, Flags: 10,
> >
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"PRIMARY","result":"true","details":"Status:
> > PRIMARY
> > ","wait":"0","bypassHostMaintenance":"false"}}] }
> > 2022-08-29 17:02:00,599 DEBUG [c.c.a.t.Request]
> > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
> > 6-4833769775052102467: Received: { Ans: , MgmtId: 90520740759984, via:
> > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer } }
> > 2022-08-29 17:02:00,599 DEBUG [c.c.a.m.AgentManagerImpl]
> > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Details
> > from executing class com.cloud.agent.api.CheckRouterCommand: Status:
> PRIMARY
> >
> > 2022-08-29 17:02:00,612 DEBUG [c.c.a.t.Request]
> > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
> > 1-5410793477308947404: Sending { Cmd , MgmtId: 90520740759984, via:
> > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
> router.name
> ":"r-80-VM","router.ip":"169.254.217.200"},"wait":"30","bypassHostMaintenance":"false"}}]
> > }
> > 2022-08-29 17:02:00,748 DEBUG [c.c.a.m.AgentManagerImpl]
> > (AgentManager-Handler-4:null) (logid:) Ping from 11(s-32-VM)
> > 2022-08-29 17:02:00,750 DEBUG [c.c.a.t.Request]
> > (AgentManager-Handler-1:null) (logid:) Seq 1-5410793477308947404:
> > Processing: { Ans: , MgmtId: 90520740759984, via: 1, Ver: v1, Flags: 10,
> >
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"BACKUP","result":"true","details":"Status:
> > BACKUP
> > ","wait":"0","bypassHostMaintenance":"false"}}] }
> > 2022-08-29 17:02:00,750 DEBUG [c.c.a.t.Request]
> > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
> > 1-5410793477308947404: Received: { Ans: , MgmtId: 90520740759984, via:
> > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer } }
> > 2022-08-29 17:02:00,751 DEBUG [c.c.a.m.AgentManagerImpl]
> > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Details
> > from executing class com.cloud.agent.api.CheckRouterCommand: Status:
> BACKUP
> >
> > 2022-08-29 17:02:01,362 DEBUG [c.c.s.StatsCollector]
> > (Cluster-Worker-839:ctx-e097a8ab) (logid:3deacf65) StatusUpdate from
> > 90520740759984, json:
> >
> {"managementServerHostId":1,"managementServerHostUuid":"5615ca5f-0716-4da1-93ec-31f79fd19bbe","collectionTime":"Aug
> > 29, 2022, 5:02:01
> >
> PM","sessions":1,"cpuUtilization":0.0,"totalJvmMemoryBytes":324534272,"freeJvmMemoryBytes":68595424,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":103760388,"jvmStartTime":1661681560910,"availableProcessors":6,"loadAverage":0.02,"totalInit":139788288,"totalUsed":474791104,"totalCommitted":557662208,"pid":1522332,"jvmName":"1522332@vm-1-cs-mgmt.aaaaaaaa.bbb
> ","jvmVendor":"Ubuntu","jvmVersion":"11.0.16+8-post-Ubuntu-0ubuntu120.04","osDistribution":"Ubuntu
> > 20.04.5
> >
> LTS","agentCount":5,"heapMemoryUsed":256775312,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":13,"threadsRunnableCount":15,"threadsTerminatedCount":0,"threadsTotalCount":420,"threadsWaitingCount":331,"systemMemoryTotal":8343236608,"systemMemoryFree":2623193088,"systemMemoryUsed":814808,"systemMemoryVirtualSize":7494184960,"logInfo":"/var/log/cloudstack/management/management-server.log
> > using: 74M\non disk /dev/vda2 mounted on / (14%
> >
> full)","systemTotalCpuCycles":15870.180000000002,"systemLoadAverages":[0.02,0.03,0.0],"systemCyclesUsage":[4116569,2615101,2716460424],"dbLocal":false,"usageLocal":false,"systemBootTime":"Jul
> > 8, 2022, 2:35:55 AM","kernelVersion":"5.4.0-121-generic"}
> > 2022-08-29 17:02:01,723 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-4:ctx-7abb1b5d) (logid:2320950d) HA health
> check
> > task is running...
> > 2022-08-29 17:02:05,417 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (Vm-Operations-Cleanup-1:ctx-e741e208) (logid:3f5129e0) VM Operation
> Thread
> > Running
> > 2022-08-29 17:02:05,725 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-4:ctx-8e7f6766) (logid:710cb3b2) HA health
> check
> > task is running...
> > 2022-08-29 17:02:08,626 DEBUG [c.c.a.m.AgentManagerImpl]
> > (AgentManager-Handler-11:null) (logid:) Ping from Routing host
> > 6(srv-3-comp.aaaaaaaa.bbb)
> >
> > 2022-08-29 17:02:09,727 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-4:ctx-26a5ff07) (logid:09b3c89a) HA health
> check
> > task is running...
> > 2022-08-29 17:02:13,729 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-4:ctx-b4d4e374) (logid:9048c7a7) HA health
> check
> > task is running...
> > 2022-08-29 17:02:17,731 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-4:ctx-3db3d208) (logid:7bb73f7b) HA health
> check
> > task is running...
> > 2022-08-29 17:02:21,734 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-1:ctx-fca1d248) (logid:1ed5381a) HA health
> check
> > task is running...
> > 2022-08-29 17:02:25,737 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-1:ctx-984e2382) (logid:e6c3d724) HA health
> check
> > task is running...
> > 2022-08-29 17:02:25,759 DEBUG [c.c.s.StatsCollector]
> > (secstorage-1:ctx-64c610ed) (logid:6603cd38) Verifying image storage [1].
> > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
> > 2022-08-29 17:02:25,763 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > (secstorage-1:ctx-64c610ed) (logid:6603cd38) Zone [1] is ready to launch
> > secondary storage VM.
> > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Zone 1 is ready to launch
> > console proxy
> > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Expand console proxy
> standby
> > capacity for zone xxxxxxxxxx
> > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Allocating console proxy
> > standby capacity for zone [1].
> > 2022-08-29 17:02:26,030 INFO [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Found a stopped console
> > proxy, starting it. Vm id : 85
> > 2022-08-29 17:02:26,031 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) start parameter value of
> > enterHardwareSetup == <very null> during processing of queued job
> > 2022-08-29 17:02:26,037 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Sync job-4289 execution on
> > object VmWorkJobQueue.85
> > 2022-08-29 17:02:26,167 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-bba6b89d) (logid:f9aed0eb) Execute
> sync-queue
> > item: SyncQueueItemVO {id:2106, queueId: 2105, contentType: AsyncJob,
> > contentId: 4289, lastProcessMsid: 90520740759984, lastprocessNumber: 2,
> > lastProcessTime: Mon Aug 29 17:02:26 CEST 2022, created: Mon Aug 29
> > 17:02:26 CEST 2022}
> > 2022-08-29 17:02:26,167 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-bba6b89d) (logid:f9aed0eb) Schedule queued
> > job-4289
> > 2022-08-29 17:02:26,172 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:f919f9b3)
> Add
> > job-4289 into job monitoring
> > 2022-08-29 17:02:26,181 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5)
> > Executing AsyncJobVO: {id:4289, userId: 1, accountId: 1, instanceType:
> > null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
> >
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAVXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
> > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> > result: null, initMsid: 90520740759984, completeMsid: null, lastUpdated:
> > null, lastPolled: null, created: Mon Aug 29 17:02:26 CEST 2022, removed:
> > null}
> > 2022-08-29 17:02:26,181 DEBUG [c.c.v.VmWorkJobDispatcher]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5)
> Run
> > VM work job: com.cloud.vm.VmWorkStart for VM 85, job origin: 3445
> > 2022-08-29 17:02:26,183 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Execute VM work job:
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> > 2022-08-29 17:02:26,184 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) orchestrating VM start for 'v-85-VM'
> > com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
> > 2022-08-29 17:02:26,192 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
> > transited from [Stopped] to [Starting] with event [StartRequested]. VM's
> > original host: null, new host: null, host before state transition: null
> > 2022-08-29 17:02:26,192 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Successfully transitioned to start state for VM instance
> > {id: "85", name: "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
> > type="ConsoleProxy"} reservation id =
> 24bf955f-96d8-4f4f-ae33-2cae2f3e0425
> > 2022-08-29 17:02:26,198 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts: null
> > 2022-08-29 17:02:26,200 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) VM start attempt #1
> > 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
> > 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Trying to allocate a host and storage pools from dc:1,
> > pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
> > 1073741824
> > 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
> > 2022-08-29 17:02:26,211 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
> > 2022-08-29 17:02:26,212 DEBUG [c.c.d.FirstFitPlanner]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Searching all possible resources under this Zone: 1
> > 2022-08-29 17:02:26,213 DEBUG [c.c.d.FirstFitPlanner]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Listing clusters in order of aggregate capacity, that
> have
> > (at least one host with) enough CPU and RAM capacity under this Zone: 1
> > 2022-08-29 17:02:26,216 DEBUG [c.c.d.FirstFitPlanner]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Removing from the clusterId list these clusters from
> avoid
> > set: []
> > 2022-08-29 17:02:26,221 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
> > 2022-08-29 17:02:26,222 INFO [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested with
> > Custom[UEFI] Boot Type false
> > 2022-08-29 17:02:26,223 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in dc: 1
> > pod:1 cluster:1
> > 2022-08-29 17:02:26,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4 hosts
> to
> > check for allocation: [Host {"id": "7", "name":
> "srv-5-comp.aaaaaaaa.bbb",
> > "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
> > {"id": "1", "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}, Host {"id":
> "6",
> > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id":
> "5",
> > "name": "srv-4-comp.aaaaaaaa.bbb", "uuid":
> > "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}]
> > 2022-08-29 17:02:26,231 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for allocation
> > after prioritization: [Host {"id": "7", "name":
> "srv-5-comp.aaaaaaaa.bbb",
> > "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
> > {"id": "1", "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}, Host {"id":
> "6",
> > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id":
> "5",
> > "name": "srv-4-comp.aaaaaaaa.bbb", "uuid":
> > "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}]
> > 2022-08-29 17:02:26,231 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for speed=500Mhz,
> > Ram=1024 MB
> > 2022-08-29 17:02:26,231 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
> > srv-5-comp.aaaaaaaa.bbb, uuid: 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is
> KVM
> > hypervisor type, no max guest limit check needed
> > 2022-08-29 17:02:26,234 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu capability
> > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> > 2022-08-29 17:02:26,234 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has enough
> > capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> > cpuOverprovisioningFactor: 1.0
> > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> > and CPU after applying overprovisioning: 127200
> > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 , Requested
> CPU:
> > 500
> > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
> > 189373743104 , Requested RAM: (1.00 GB) 1073741824
> > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > available
> > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> > 7, used: 90100, reserved: 0, actual total: 127200, total with
> > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > ,considerReservedCapacity?: true
> > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> > 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total: (250.27
> GB)
> > 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> > false , considerReservedCapacity?: true
> > 2022-08-29 17:02:26,236 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding
> to
> > list: 7
> > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
> > srv-2-comp.aaaaaaaa.bbb, uuid: 883f7f62-55f2-42f4-8242-16b6847c4204} is
> KVM
> > hypervisor type, no max guest limit check needed
> > 2022-08-29 17:02:26,239 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu capability
> > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> > 2022-08-29 17:02:26,239 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has enough
> > capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> > cpuOverprovisioningFactor: 1.0
> > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> > and CPU after applying overprovisioning: 127200
> > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 , Requested
> CPU:
> > 500
> > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
> > 229802639360 , Requested RAM: (1.00 GB) 1073741824
> > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > available
> > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> > 1, used: 48200, reserved: 0, actual total: 127200, total with
> > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > ,considerReservedCapacity?: true
> > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> > 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total: (250.27
> GB)
> > 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> > false , considerReservedCapacity?: true
> > 2022-08-29 17:02:26,240 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding
> to
> > list: 1
> > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
> > srv-3-comp.aaaaaaaa.bbb, uuid: 6858713b-7e17-46d8-854d-24a13f9c7001} is
> KVM
> > hypervisor type, no max guest limit check needed
> > 2022-08-29 17:02:26,243 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu capability
> > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> > 2022-08-29 17:02:26,243 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has enough
> > capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> > cpuOverprovisioningFactor: 1.0
> > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> > and CPU after applying overprovisioning: 127200
> > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 , Requested
> CPU:
> > 500
> > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
> > 204032835584 , Requested RAM: (1.00 GB) 1073741824
> > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > available
> > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> > 6, used: 42900, reserved: 0, actual total: 127200, total with
> > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > ,considerReservedCapacity?: true
> > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> > 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total: (250.27
> GB)
> > 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> > false , considerReservedCapacity?: true
> > 2022-08-29 17:02:26,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding
> to
> > list: 6
> > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
> > srv-4-comp.aaaaaaaa.bbb, uuid: 134c625d-15ab-461a-a38d-36fb0b979b5d} is
> KVM
> > hypervisor type, no max guest limit check needed
> > 2022-08-29 17:02:26,247 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu capability
> > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> > 2022-08-29 17:02:26,247 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has enough
> > capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> > cpuOverprovisioningFactor: 1.0
> > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> > and CPU after applying overprovisioning: 127200
> > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 , Requested
> > CPU: 500
> > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
> > 251009040384 , Requested RAM: (1.00 GB) 1073741824
> > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > available
> > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> > 5, used: 11100, reserved: 0, actual total: 127200, total with
> > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > ,considerReservedCapacity?: true
> > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> > 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total: (250.27
> GB)
> > 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> > false , considerReservedCapacity?: true
> > 2022-08-29 17:02:26,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding
> to
> > list: 5
> > 2022-08-29 17:02:26,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator returning 4
> > suitable hosts
> > 2022-08-29 17:02:26,260 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
> > transited from [Starting] to [Stopped] with event [OperationFailed]. VM's
> > original host: null, new host: null, host before state transition: null
> > 2022-08-29 17:02:26,260 WARN [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Unable to orchestrate start VM instance {id: "85", name:
> > "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
> > type="ConsoleProxy"} due to [Unable to create deployment, no usable
> volumes
> > found for the VM: 85].
> > com.cloud.utils.exception.CloudRuntimeException: Unable to create
> > deployment, no usable volumes found for the VM: 85
> > at
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
> > at
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
> > at
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
> > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
> > at
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > at java.base/java.lang.Thread.run(Thread.java:829)
> > 2022-08-29 17:02:26,261 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Done executing VM work job:
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> > 2022-08-29 17:02:26,262 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Complete async job-4289, jobStatus: FAILED, resultCode:
> 0,
> > result:
> >
> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODUiLCBuYW1lOiAidi04NS1WTSIsIHV1aWQ6ICIxMTBkODYyMC1jMDI5LTRjYTktOTA4NC0yMTAwOTk3MjVhODAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODVdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
> > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Publish async job-4289 complete on message bus
> > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Wake up jobs related to job-4289
> > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Update db status for job-4289
> > 2022-08-29 17:02:26,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Wake up jobs joined with job-4289 and disjoin all
> subjobs
> > created from job- 4289
> > 2022-08-29 17:02:26,269 DEBUG [c.c.v.VmWorkJobDispatcher]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5)
> > Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 85, job
> > origin: 3445
> > 2022-08-29 17:02:26,270 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5)
> > Done executing com.cloud.vm.VmWorkStart for job-4289
> > 2022-08-29 17:02:26,271 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5)
> > Remove job-4289 from job monitoring
> > 2022-08-29 17:02:26,277 WARN [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Unable to allocate console
> > proxy standby capacity for zone [1] due to [Unable to orchestrate start
> VM
> > instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> [Unable
> > to create deployment, no usable volumes found for the VM: 85].].
> > com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate
> > start VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> [Unable
> > to create deployment, no usable volumes found for the VM: 85].
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
> > at
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > at java.base/java.lang.Thread.run(Thread.java:829)
> > 2022-08-29 17:02:26,277 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) received console proxy
> alert
> >
> >
> >
> > Am Mo., 29. Aug. 2022 um 16:30 Uhr schrieb Slavka Peleva <
> > slavkap@storpool.com>:
> >
> >> Thanks, Chris, for the information!
> >>
> >> Are you using custom offerings for the system VMs? Can you share more
> >> info if yes?
> >>
> >> About the log, it's not the complete log from where the deployment
> >> started. The deployment begins a bit earlier than what you shared. Can
> you
> >> search the management log for `Sync job-3539 execution on object
> >> VmWorkJobQueue` to get the `logid` for the volume creation (coloured in
> >> blue)? And if possible, share the logs with the found log ID
> >>
> >> Here is an example from my logs:
> >>
> >> 2022-08-29 17:07:49,114 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (consoleproxy-1:ctx-a952a3dc) (logid:89157bb9) Sync job-60484 execution
> >> on object VmWorkJobQueue.4904
> >>
> >> 2022-08-29 17:07:50,816 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> >> (Work-Job-Executor-5:ctx-56172581 job-60479/job-60484 ctx-a900986d)
> >> (logid:95d750ae) Execute VM work job:
> >>
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":4904,"handlerName":"VirtualMachineManagerImpl"}
> >> 2022-08-29 17:07:50,817 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >> (Work-Job-Executor-5:ctx-56172581 job-60479/job-60484 ctx-a900986d)
> >> (logid:95d750ae) orchestrating VM start for 'v-4904-VM'
> >> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
> >>
> >>
> >> On Mon, Aug 29, 2022 at 2:29 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
> >>
> >>> Hi Slavka,
> >>>
> >>> thats what i did after your suggestion / informatin and where the error
> >>> occured.
> >>> Deleted / Destroyed the consoleproxy and waited till recreation. Which
> >>> didn't worked throwing the error i pasted above.
> >>> Sorry for beeing so unspecific.
> >>>
> >>> Regards,
> >>> Chris
> >>>
> >>> Am Mo., 29. Aug. 2022 um 12:46 Uhr schrieb Slavka Peleva <
> >>> slavkap@storpool.com>:
> >>>
> >>>> Hi Chris,
> >>>>
> >>>> I mean to recreate the system VMs by destroying them.
> >>>>
> >>>> On Mon, Aug 29, 2022 at 12:38 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
> >>>>
> >>>>> Hi Slavka,
> >>>>>
> >>>>> didn't tried to restart the SystemVMs. Reading through the docs i had
> >>>>> the impression that "live-patching" would be enough for this....
> >>>>> Anyway - I tried my luck with the console proxy.
> >>>>>
> >>>>> Which failed utterly. Not starting as it isn't finding any useable
> >>>>> storage pool. - logs at the end of the mail.
> >>>>>
> >>>>> I tried deploying a VM using the same storage pool which worked
> >>>>> wonderfully.
> >>>>> There i noticed that all my custome storage-offerings are unuseable
> at
> >>>>> the moment. They are not accepting any custome storage sizes
> anymore...
> >>>>> Throwing failures "VM Creation failed. Volume size: -1GB is out of
> allowed
> >>>>> tange. Max:3072 Min:1"...
> >>>>> So this is also currently broken....
> >>>>>
> >>>>> 2022-08-29 10:51:56,189 DEBUG [c.c.v.VmWorkJobDispatcher]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
> (logid:ba10fec5) Run
> >>>>> VM work job: com.cloud.vm.VmWorkStart for VM 83, job origin: 3445
> >>>>> 2022-08-29 10:51:56,191 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Execute VM work job:
> >>>>>
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
> >>>>> 2022-08-29 10:51:56,192 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) orchestrating VM start for 'v-83-VM'
> >>>>> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
> >>>>> 2022-08-29 10:51:56,198 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
> >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} state
> >>>>> transited from [Stopped] to [Starting] with event [StartRequested].
> VM's
> >>>>> original host: null, new host: null, host before state transition:
> null
> >>>>> 2022-08-29 10:51:56,198 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Successfully transitioned to start state for VM
> instance
> >>>>> {id: "83", name: "v-83-VM", uuid:
> "b1d1f9c6-eafe-4527-9335-e649c646aab0",
> >>>>> type="ConsoleProxy"} reservation id =
> 6641ccc0-bcef-4b1a-a1f7-8dd8000626ad
> >>>>> 2022-08-29 10:51:56,203 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts:
> null
> >>>>> 2022-08-29 10:51:56,206 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) VM start attempt #1
> >>>>> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
> >>>>> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Trying to allocate a host and storage pools from
> dc:1,
> >>>>> pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
> >>>>> 1073741824
> >>>>> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
> >>>>> 2022-08-29 10:51:56,214 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
> >>>>> 2022-08-29 10:51:56,215 DEBUG [c.c.d.FirstFitPlanner]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Searching all possible resources under this Zone: 1
> >>>>> 2022-08-29 10:51:56,216 DEBUG [c.c.d.FirstFitPlanner]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Listing clusters in order of aggregate capacity,
> that have
> >>>>> (at least one host with) enough CPU and RAM capacity under this
> Zone: 1
> >>>>> 2022-08-29 10:51:56,219 DEBUG [c.c.d.FirstFitPlanner]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Removing from the clusterId list these clusters
> from avoid
> >>>>> set: []
> >>>>> 2022-08-29 10:51:56,224 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
> >>>>> 2022-08-29 10:51:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested
> with
> >>>>> Custom[UEFI] Boot Type false
> >>>>> 2022-08-29 10:51:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in dc: 1
> >>>>> pod:1 cluster:1
> >>>>> 2022-08-29 10:51:56,227 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4
> hosts to
> >>>>> check for allocation: [Host {"id": "7", "name":
> "srv-5-comp.blabla.blub",
> >>>>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"},
> Host
> >>>>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
> >>>>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
> {"id": "6",
> >>>>> "name": "srv-3-comp.blabla.blub", "uuid":
> >>>>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
> {"id": "1",
> >>>>> "name": "srv-2-comp.blabla.blub", "uuid":
> >>>>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for
> allocation
> >>>>> after prioritization: [Host {"id": "7", "name":
> "srv-5-comp.blabla.blub",
> >>>>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"},
> Host
> >>>>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
> >>>>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
> {"id": "6",
> >>>>> "name": "srv-3-comp.blabla.blub", "uuid":
> >>>>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
> {"id": "1",
> >>>>> "name": "srv-2-comp.blabla.blub", "uuid":
> >>>>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for speed=500Mhz,
> >>>>> Ram=1024 MB
> >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
> >>>>> srv-5-comp.blabla.blub, uuid: 8c753ce1-0253-49c9-9c7f-1f971a719ce6}
> is KVM
> >>>>> hypervisor type, no max guest limit check needed
> >>>>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu capability
> >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
> speed: 500
> >>>>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has
> enough
> >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> 1073741824 ,
> >>>>> cpuOverprovisioningFactor: 1.0
> >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> >>>>> and CPU after applying overprovisioning: 127200
> >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 ,
> Requested CPU:
> >>>>> 500
> >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
> >>>>> 189373743104 , Requested RAM: (1.00 GB) 1073741824
> >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
> RAM
> >>>>> available
> >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> >>>>> 7, used: 90100, reserved: 0, actual total: 127200, total with
> >>>>> overprovisioning: 127200; requested
> cpu:500,alloc_from_last_host?:false
> >>>>> ,considerReservedCapacity?: true
> >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> >>>>> 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total:
> (250.27 GB)
> >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
> alloc_from_last_host?:
> >>>>> false , considerReservedCapacity?: true
> >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> adding to
> >>>>> list: 7
> >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
> >>>>> srv-4-comp.blabla.blub, uuid: 134c625d-15ab-461a-a38d-36fb0b979b5d}
> is KVM
> >>>>> hypervisor type, no max guest limit check needed
> >>>>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu capability
> >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
> speed: 500
> >>>>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has
> enough
> >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> 1073741824 ,
> >>>>> cpuOverprovisioningFactor: 1.0
> >>>>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> >>>>> and CPU after applying overprovisioning: 127200
> >>>>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 ,
> Requested
> >>>>> CPU: 500
> >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
> >>>>> 251009040384 , Requested RAM: (1.00 GB) 1073741824
> >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
> RAM
> >>>>> available
> >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> >>>>> 5, used: 11100, reserved: 0, actual total: 127200, total with
> >>>>> overprovisioning: 127200; requested
> cpu:500,alloc_from_last_host?:false
> >>>>> ,considerReservedCapacity?: true
> >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> >>>>> 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total:
> (250.27 GB)
> >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
> alloc_from_last_host?:
> >>>>> false , considerReservedCapacity?: true
> >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> adding to
> >>>>> list: 5
> >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
> >>>>> srv-3-comp.blabla.blub, uuid: 6858713b-7e17-46d8-854d-24a13f9c7001}
> is KVM
> >>>>> hypervisor type, no max guest limit check needed
> >>>>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu capability
> >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
> speed: 500
> >>>>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has
> enough
> >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> 1073741824 ,
> >>>>> cpuOverprovisioningFactor: 1.0
> >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> >>>>> and CPU after applying overprovisioning: 127200
> >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 ,
> Requested CPU:
> >>>>> 500
> >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
> >>>>> 204032835584 , Requested RAM: (1.00 GB) 1073741824
> >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
> RAM
> >>>>> available
> >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> >>>>> 6, used: 42900, reserved: 0, actual total: 127200, total with
> >>>>> overprovisioning: 127200; requested
> cpu:500,alloc_from_last_host?:false
> >>>>> ,considerReservedCapacity?: true
> >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> >>>>> 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total:
> (250.27 GB)
> >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
> alloc_from_last_host?:
> >>>>> false , considerReservedCapacity?: true
> >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> adding to
> >>>>> list: 6
> >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
> >>>>> srv-2-comp.blabla.blub, uuid: 883f7f62-55f2-42f4-8242-16b6847c4204}
> is KVM
> >>>>> hypervisor type, no max guest limit check needed
> >>>>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu capability
> >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
> speed: 500
> >>>>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has
> enough
> >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> 1073741824 ,
> >>>>> cpuOverprovisioningFactor: 1.0
> >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> >>>>> and CPU after applying overprovisioning: 127200
> >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 ,
> Requested CPU:
> >>>>> 500
> >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
> >>>>> 229802639360 , Requested RAM: (1.00 GB) 1073741824
> >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
> RAM
> >>>>> available
> >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> >>>>> 1, used: 48200, reserved: 0, actual total: 127200, total with
> >>>>> overprovisioning: 127200; requested
> cpu:500,alloc_from_last_host?:false
> >>>>> ,considerReservedCapacity?: true
> >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> >>>>> 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total:
> (250.27 GB)
> >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
> alloc_from_last_host?:
> >>>>> false , considerReservedCapacity?: true
> >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> adding to
> >>>>> list: 1
> >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator returning 4
> >>>>> suitable hosts
> >>>>> 2022-08-29 10:51:56,260 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
> >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} state
> >>>>> transited from [Starting] to [Stopped] with event [OperationFailed].
> VM's
> >>>>> original host: null, new host: null, host before state transition:
> null
> >>>>> 2022-08-29 10:51:56,260 WARN [c.c.v.VirtualMachineManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Unable to orchestrate start VM instance {id: "83",
> name:
> >>>>> "v-83-VM", uuid: "b1d1f9c6-eafe-4527-9335-e649c646aab0",
> >>>>> type="ConsoleProxy"} due to [Unable to create deployment, no usable
> volumes
> >>>>> found for the VM: 83].
> >>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to create
> >>>>> deployment, no usable volumes found for the VM: 83
> >>>>> at
> >>>>>
> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
> >>>>> at
> >>>>>
> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
> >>>>> at
> >>>>>
> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
> >>>>> at
> >>>>>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
> >>>>> at
> >>>>>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
> >>>>> at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> >>>>> Source)
> >>>>> at
> >>>>>
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >>>>> at
> >>>>>
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> >>>>> at
> >>>>>
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> >>>>> at
> >>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> >>>>> at
> >>>>>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> >>>>> at
> >>>>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> >>>>> at
> >>>>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> >>>>> at
> >>>>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> >>>>> at
> >>>>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> >>>>> at
> >>>>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> >>>>> at
> >>>>>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> >>>>> at
> >>>>>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> >>>>> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >>>>> at
> >>>>>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >>>>> at
> >>>>>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >>>>> at java.base/java.lang.Thread.run(Thread.java:829)
> >>>>> 2022-08-29 10:51:56,262 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Done executing VM work job:
> >>>>>
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
> >>>>> 2022-08-29 10:51:56,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Complete async job-3539, jobStatus: FAILED,
> resultCode: 0,
> >>>>> result:
> >>>>>
> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODMiLCBuYW1lOiAidi04My1WTSIsIHV1aWQ6ICJiMWQxZjljNi1lYWZlLTQ1MjctOTMzNS1lNjQ5YzY0NmFhYjAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODNdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
> >>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Publish async job-3539 complete on message bus
> >>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Wake up jobs related to job-3539
> >>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Update db status for job-3539
> >>>>> 2022-08-29 10:51:56,266 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Wake up jobs joined with job-3539 and disjoin all
> subjobs
> >>>>> created from job- 3539
> >>>>> 2022-08-29 10:51:56,276 DEBUG [c.c.v.VmWorkJobDispatcher]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
> (logid:ba10fec5) Done
> >>>>> with run of VM work job: com.cloud.vm.VmWorkStart for VM 83, job
> origin:
> >>>>> 3445
> >>>>> 2022-08-29 10:51:56,276 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
> (logid:ba10fec5) Done
> >>>>> executing com.cloud.vm.VmWorkStart for job-3539
> >>>>> 2022-08-29 10:51:56,278 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
> (logid:ba10fec5)
> >>>>> Remove job-3539 from job monitoring
> >>>>> 2022-08-29 10:51:56,285 WARN [c.c.c.ConsoleProxyManagerImpl]
> >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Unable to allocate
> console
> >>>>> proxy standby capacity for zone [1] due to [Unable to orchestrate
> start VM
> >>>>> instance {id: "83", name: "v-83-VM", uuid:
> >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to
> [Unable
> >>>>> to create deployment, no usable volumes found for the VM: 83].].
> >>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to
> orchestrate
> >>>>> start VM instance {id: "83", name: "v-83-VM", uuid:
> >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to
> [Unable
> >>>>> to create deployment, no usable volumes found for the VM: 83].
> >>>>> at
> >>>>>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> >>>>> at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> >>>>> Source)
> >>>>> at
> >>>>>
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >>>>> at
> >>>>>
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> >>>>> at
> >>>>>
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> >>>>> at
> >>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> >>>>> at
> >>>>>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> >>>>> at
> >>>>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> >>>>> at
> >>>>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> >>>>> at
> >>>>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> >>>>> at
> >>>>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> >>>>> at
> >>>>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> >>>>> at
> >>>>>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> >>>>> at
> >>>>>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> >>>>> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >>>>> at
> >>>>>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >>>>> at
> >>>>>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >>>>> at java.base/java.lang.Thread.run(Thread.java:829)
> >>>>> 2022-08-29 10:51:56,287 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) received console
> proxy alert
> >>>>> 2022-08-29 10:51:56,288 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Console proxy creation
> >>>>> failure, zone: blabla
> >>>>> 2022-08-29 10:51:56,289 WARN [c.c.a.AlertManagerImpl]
> >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) alertType=[10]
> >>>>> dataCenterId=[1] podId=[null] clusterId=[null] message=[Console proxy
> >>>>> creation failure. zone: blabla, error details: Unable to orchestrate
> start
> >>>>> VM instance {id: "83", name: "v-83-VM", uuid:
> >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to
> [Unable
> >>>>> to create deployment, no usable volumes found for the VM: 83].].
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> Am Mo., 29. Aug. 2022 um 09:40 Uhr schrieb Slavka Peleva
> >>>>> <sl...@storpool.com.invalid>:
> >>>>>
> >>>>>> Hi Chris,
> >>>>>>
> >>>>>> Did you recreate the system VMs? In the 4.17 version, the
> >>>>>> `systemvm.iso` is
> >>>>>> deprecated. In its place, you should have the file on agents
> >>>>>> `/usr/share/cloudstack-common/vms/agent.zip`.
> >>>>>> Can you share the complete log if the system VMs are newly created
> >>>>>> after
> >>>>>> the upgrade?
> >>>>>>
> >>>>>> Best regards,
> >>>>>> Slavka
> >>>>>>
> >>>>>> On Sun, Aug 28, 2022 at 2:16 PM Vash_X@gmx.de <Va...@gmx.de>
> wrote:
> >>>>>>
> >>>>>> > Hi everyone,
> >>>>>> >
> >>>>>> > faceing some challanges again after upgradeing to 4.17.
> >>>>>> > Did as explained in the docs.
> >>>>>> >
> >>>>>> > Afterwards I am now not able to migrate system VMs to different
> >>>>>> hosts.
> >>>>>> > Getting the error
> >>>>>> > "Exception during migrate: org.libvirt.LibvirtException: Cannot
> >>>>>> access
> >>>>>> > storage file '/usr/share/cloudstack-common/vms/systemvm.iso': No
> >>>>>> such file
> >>>>>> > or directory"
> >>>>>> >
> >>>>>> > Checked the exitstence of the file on the servers:
> >>>>>> > Mangement-Server has a "systemvm.iso.bak"
> >>>>>> > Hosts - no "systemvm.iso" at all present.
> >>>>>> >
> >>>>>> > Maybe someone has an idea on how to move on with this?
> >>>>>> >
> >>>>>> > Regards,
> >>>>>> > Chris
> >>>>>> >
> >>>>>>
> >>>>>
>
Re: Error migrate System VMs after upgrageing to 4.17
Posted by "Vash_X@gmx.de" <Va...@gmx.de>.
Hi Slavka,
that was my first thought, too. I checked beforehand and confirmed with
your provided SQL Statements:
*`select * from user_ip_address where forsystemvms = 1;`* <---- returns my
defined (Public) IPs out of the correct range for SystemVMs
*`SELECT * FROM clou*d.user_ip_address where allocated IS NULL AND removed
IS NULL; <-- All IPs shown beside the one which is currently in use for the
running ssvm. Meaning from my point of view "free" for allocation for other
system VMs.
Querys performed directly on the DB-Server.
I can't see a reason why there shouldn't be any IPs left for allocation.
Also it is interesting why the "reasoning" for the failing deployment
changes from "no IP availeable" to "no useable volumes found".
How ever i can't see anything missing at the moment, what makes me even
more puzzeled.
Regards
Chris
Am Di., 30. Aug. 2022 um 13:31 Uhr schrieb Slavka Peleva
<sl...@storpool.com.invalid>:
> Hi Chris,
>
> The deployment fails because CS cannot find a free IP address to allocate
> for the system VM. Can you check if you explicitly added a range for the
> system VMs, or you can check with this `select * from user_ip_address where
> forsystemvms = 1;` If you don't have any ranges, you can get free IPs
> with `SELECT
> * FROM cloud.user_ip_address where allocated IS NULL AND removed IS NULL;`
> . To get the exact query executed by CS, you need to increase the log level
> to TRACE.
>
> Best regards,
> Slavka
>
> On Mon, Aug 29, 2022 at 8:03 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
>
> > Hi Slavka,
> >
> > no, i am not using any kind of custome offerings.
> >
> > I try to provide a more detailled log this time.
> > Situation:
> > v-83-VM was the "broken" consoleproxy. So the following log starts with
> > it's expunging.
> > Afterwards the new consoleproxy shall be deployed - v-85-VM.
> >
> > This first deployment (logid:dcfe0999) fails due to Insufficent address
> > capacity - which i can't really understand or see in the database,
> checking
> > the availeable IPs for systemVMs (given that i might looking up the wrong
> > tables....).
> > How ever CS is working as expected trying to deploy the console proxy
> > (logid:440fd101) Sync job-4288).
> > That job fails with "Unable to create deployment, no usable volumes found
> > for the VM: 85".
> > Afterwards i have another try to start the consoleproxy (job-4289) which
> > fails for the same reason. And thats how things go on in the log file...
> >
> >
> > 2022-08-29 17:01:16,224 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> > Cleaning up hypervisor data structures (ex. SRs in XenServer) for managed
> > storage. Data from VM instance {id: "83", name: "v-83-VM", uuid:
> > "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}.
> > 2022-08-29 17:01:16,225 DEBUG [o.a.c.e.o.VolumeOrchestrator]
> > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> > Cleaning storage for vm: 83
> > 2022-08-29 17:01:16,233 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> > Expunged VM instance {id: "83", name: "v-83-VM", uuid:
> > "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
> > 2022-08-29 17:01:16,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> > Complete async job-4286, jobStatus: SUCCEEDED, resultCode: 0, result:
> >
> org.apache.cloudstack.api.response.SystemVmResponse/systemvm/{"id":"b1d1f9c6-eafe-4527-9335-e649c646aab0","systemvmtype":"consoleproxy","zoneid":"48e2e928-3300-43b5-8e3a-d9b358022502","zonename":"xxxxxxxxxx","dns1":"10.129.0.4","dns2":"195.10.195.195","name":"v-83-VM","templateid":"c20799e4-a990-449a-aae6-4b67374c7aff","templatename":"systemvm-kvm-4.17.0","created":"2022-08-29T10:38:26+0200","state":"Stopped","isdynamicallyscalable":"false","hasannotations":"false"}
> > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> > Publish async job-4286 complete on message bus
> > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> > Wake up jobs related to job-4286
> > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> > Update db status for job-4286
> > 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> > Wake up jobs joined with job-4286 and disjoin all subjobs created from
> job-
> > 4286
> > 2022-08-29 17:01:16,260 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (API-Job-Executor-35:ctx-29e06f28 job-4286) (logid:5fe8559a) Done
> executing
> > org.apache.cloudstack.api.command.admin.systemvm.DestroySystemVmCmd for
> > job-4286
> > 2022-08-29 17:01:16,260 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > (API-Job-Executor-35:ctx-29e06f28 job-4286) (logid:5fe8559a) Remove
> > job-4286 from job monitoring
> > 2022-08-29 17:01:17,691 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-6:ctx-6af18142) (logid:fe263a70) HA health
> check
> > task is running...
> >
> > 2022-08-29 17:01:21,694 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-6:ctx-89b1442f) (logid:0efc40b2) HA health
> check
> > task is running...
> > 2022-08-29 17:01:25,698 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-5:ctx-62313d05) (logid:7073663f) HA health
> check
> > task is running...
> > 2022-08-29 17:01:25,767 DEBUG [c.c.s.StatsCollector]
> > (secstorage-1:ctx-c80e5a6e) (logid:f8e3eb60) Verifying image storage [1].
> > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
> > 2022-08-29 17:01:25,773 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > (secstorage-1:ctx-c80e5a6e) (logid:f8e3eb60) Zone [1] is ready to launch
> > secondary storage VM.
> > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Zone 1 is ready to launch
> > console proxy
> > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Expand console proxy
> standby
> > capacity for zone xxxxxxxxxx
> > 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating console proxy
> > standby capacity for zone [1].
> > 2022-08-29 17:01:26,032 INFO [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) No stopped console proxy
> is
> > available, need to allocate a new console proxy
> > 2022-08-29 17:01:26,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Assign console proxy from
> a
> > newly started instance for request from data center : 1
> > 2022-08-29 17:01:26,056 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing network
> > configuration for offering [Network Offering
> > [1-Public-System-Public-Network]: Network {"id": 200, "name": "null",
> > "uuid": "1b0bf965-a391-478d-833d-3e37e4a15eb8", "networkofferingid": 1}
> > 2022-08-29 17:01:26,056 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
> > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account {"id": 1,
> > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
> > 2022-08-29 17:01:26,064 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing network
> > configuration for offering [Network Offering
> > [3-Control-System-Control-Network]: Network {"id": 202, "name": "null",
> > "uuid": "2af42518-fbf4-4a35-9d6b-370e6ad315c7", "networkofferingid": 3}
> > 2022-08-29 17:01:26,064 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
> > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account {"id": 1,
> > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
> > 2022-08-29 17:01:26,069 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing network
> > configuration for offering [Network Offering
> > [2-Management-System-Management-Network]: Network {"id": 201, "name":
> > "null", "uuid": "53711d87-a3c6-4f33-bfce-c30e0de72760",
> > "networkofferingid": 2}
> > 2022-08-29 17:01:26,069 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
> > Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account {"id": 1,
> > "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
> > 2022-08-29 17:01:26,077 INFO [c.c.v.VirtualMachineManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) allocating virtual machine
> > from template:c20799e4-a990-449a-aae6-4b67374c7aff with hostname:v-85-VM
> > and 3 networks
> > 2022-08-29 17:01:26,079 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating entries for VM:
> > VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
> > 2022-08-29 17:01:26,082 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating nics for VM
> > instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
> > 2022-08-29 17:01:26,083 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating nic for vm VM
> > instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} in network
> > Network {"id": 200, "name": "null", "uuid":
> > "1b0bf965-a391-478d-833d-3e37e4a15eb8", "networkofferingid": 1} with
> > requested profile NicProfile
> >
> {"broadcastUri":null,"iPv4Address":null,"id":0,"reservationId":null,"vmId":0}
> > 2022-08-29 17:01:26,089 WARN [c.c.n.IpAddressManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to get ip address
> in
> > zone id=1, vlanId id=[3, 10, 13, 14]
> > 2022-08-29 17:01:26,090 DEBUG [c.c.u.d.T.Transaction]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Rolling back the
> > transaction: Time = 8 Name = consoleproxy-1; called by
> >
> -TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-Transaction.execute:38-IpAddressManagerImpl.listAvailablePublicIps:824-IpAddressManagerImpl.fetchNewPublicIp:811-IpAddressManagerImpl.fetchNewPublicIp:803-IpAddressManagerImpl.assignPublicIpAddress:784-PublicNetworkGuru.getIp:125-PublicNetworkGuru.allocate:172-NetworkOrchestrator.allocateNic:973-NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice:823
> > 2022-08-29 17:01:26,090 WARN [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to allocate proxy
> > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
> > due to [Insufficient address capacity].
> > com.cloud.exception.InsufficientAddressCapacityException: Insufficient
> > address capacityScope=interface com.cloud.dc.DataCenter; id=1
> > at
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
> > at
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
> > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> > at
> >
> com.cloud.network.IpAddressManagerImpl.listAvailablePublicIps(IpAddressManagerImpl.java:824)
> > at
> >
> com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl.java:811)
> > at
> >
> com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl.java:803)
> > at
> >
> com.cloud.network.IpAddressManagerImpl.assignPublicIpAddress(IpAddressManagerImpl.java:784)
> > at
> >
> com.cloud.network.guru.PublicNetworkGuru.getIp(PublicNetworkGuru.java:125)
> > at
> >
> com.cloud.network.guru.PublicNetworkGuru.allocate(PublicNetworkGuru.java:172)
> > at
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocateNic(NetworkOrchestrator.java:973)
> > at
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice(NetworkOrchestrator.java:823)
> > at
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$3.doInTransactionWithoutResult(NetworkOrchestrator.java:797)
> > at
> >
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
> > at
> >
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
> > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> > at
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocate(NetworkOrchestrator.java:778)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl$1.doInTransactionWithoutResult(VirtualMachineManagerImpl.java:478)
> > at
> >
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
> > at
> >
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
> > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:469)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:528)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:700)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
> > at
> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
> > at
> > com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
> > at
> >
> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > at
> >
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> > at
> >
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > at java.base/java.lang.Thread.run(Thread.java:829)
> > 2022-08-29 17:01:26,091 WARN [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to allocate console
> > proxy standby capacity for zone [1] due to [Unable to allocate proxy
> > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
> > due to [Insufficient address capacity].].
> > com.cloud.utils.exception.CloudRuntimeException: Unable to allocate proxy
> > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
> > due to [Insufficient address capacity].
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:704)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
> > at
> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
> > at
> > com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
> > at
> >
> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > at
> >
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> > at
> >
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > at java.base/java.lang.Thread.run(Thread.java:829)
> > Caused by: com.cloud.exception.InsufficientAddressCapacityException:
> > Insufficient address capacityScope=interface com.cloud.dc.DataCenter;
> id=1
> > at
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
> > at
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
> > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> > ... 38 more
> > 2022-08-29 17:01:26,091 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) received console proxy
> alert
> > 2022-08-29 17:01:26,092 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Console proxy creation
> > failure, zone: xxxxxxxxxx
> > 2022-08-29 17:01:26,093 WARN [c.c.a.AlertManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) alertType=[10]
> > dataCenterId=[1] podId=[null] clusterId=[null] message=[Console proxy
> > creation failure. zone: xxxxxxxxxx, error details: Unable to allocate
> proxy
> > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
> > due to [Insufficient address capacity].].
> > 2022-08-29 17:01:26,096 WARN [c.c.a.AlertManagerImpl]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) No recipients set in
> global
> > setting 'alert.email.addresses', skipping sending alert with subject
> > [Console proxy creation failure. zone: xxxxxxxxxx, error details: Unable
> to
> > allocate proxy [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
> > due to [Insufficient address capacity].] and content [Console proxy
> > creation failure (zone xxxxxxxxxx)].
> > 2022-08-29 17:01:26,096 WARN [c.c.v.SystemVmLoadScanner]
> > (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unexpected exception
> Unable
> > to allocate proxy [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
> > due to [Insufficient address capacity].
> > com.cloud.utils.exception.CloudRuntimeException: Unable to allocate proxy
> > [Console VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
> > due to [Insufficient address capacity].
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:704)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
> > at
> >
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
> > at
> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
> > at
> > com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
> > at
> >
> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > at
> >
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> > at
> >
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > at java.base/java.lang.Thread.run(Thread.java:829)
> > Caused by: com.cloud.exception.InsufficientAddressCapacityException:
> > Insufficient address capacityScope=interface com.cloud.dc.DataCenter;
> id=1
> > at
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
> > at
> >
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
> > at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> > ... 38 more
> > 2022-08-29 17:01:29,700 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-5:ctx-994465bd) (logid:1e2822fd) HA health
> check
> > task is running...
> > 2022-08-29 17:01:30,444 DEBUG
> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 2 routers to
> > update status.
> > 2022-08-29 17:01:30,449 DEBUG
> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 1 VPC's to
> > update Redundant State.
> > 2022-08-29 17:01:30,451 DEBUG
> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 0 networks to
> > update RvR status.
> > 2022-08-29 17:01:30,459 DEBUG [c.c.a.t.Request]
> > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
> > 6-4833769775052102463: Sending { Cmd , MgmtId: 90520740759984, via:
> > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
> router.name
> ":"r-79-VM","router.ip":"169.254.144.198"},"wait":"30","bypassHostMaintenance":"false"}}]
> > }
> > 2022-08-29 17:01:30,643 DEBUG [c.c.a.t.Request]
> > (AgentManager-Handler-12:null) (logid:) Seq 6-4833769775052102463:
> > Processing: { Ans: , MgmtId: 90520740759984, via: 6, Ver: v1, Flags: 10,
> >
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"PRIMARY","result":"true","details":"Status:
> > PRIMARY
> > ","wait":"0","bypassHostMaintenance":"false"}}] }
> > 2022-08-29 17:01:30,643 DEBUG [c.c.a.t.Request]
> > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
> > 6-4833769775052102463: Received: { Ans: , MgmtId: 90520740759984, via:
> > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer } }
> > 2022-08-29 17:01:30,643 DEBUG [c.c.a.m.AgentManagerImpl]
> > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Details
> from
> > executing class com.cloud.agent.api.CheckRouterCommand: Status: PRIMARY
> >
> > 2022-08-29 17:01:30,655 DEBUG [c.c.a.t.Request]
> > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
> > 1-5410793477308947401: Sending { Cmd , MgmtId: 90520740759984, via:
> > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
> router.name
> ":"r-80-VM","router.ip":"169.254.217.200"},"wait":"30","bypassHostMaintenance":"false"}}]
> > }
> > 2022-08-29 17:01:30,796 DEBUG [c.c.a.t.Request]
> > (AgentManager-Handler-7:null) (logid:) Seq 1-5410793477308947401:
> > Processing: { Ans: , MgmtId: 90520740759984, via: 1, Ver: v1, Flags: 10,
> >
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"BACKUP","result":"true","details":"Status:
> > BACKUP
> > ","wait":"0","bypassHostMaintenance":"false"}}] }
> > 2022-08-29 17:01:30,796 DEBUG [c.c.a.t.Request]
> > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
> > 1-5410793477308947401: Received: { Ans: , MgmtId: 90520740759984, via:
> > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer } }
> > 2022-08-29 17:01:30,796 DEBUG [c.c.a.m.AgentManagerImpl]
> > (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Details
> from
> > executing class com.cloud.agent.api.CheckRouterCommand: Status: BACKUP
> >
> > 2022-08-29 17:01:33,703 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-5:ctx-70bca202) (logid:b332dbe0) HA health
> check
> > task is running...
> > 2022-08-29 17:01:34,083 DEBUG [c.c.s.StatsCollector]
> > (StatsCollector-6:ctx-213e2ecc) (logid:aa0584c9) AutoScaling Monitor is
> > running...
> > 2022-08-29 17:01:37,707 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-6:ctx-57493517) (logid:5093e75f) HA health
> check
> > task is running...
> > 2022-08-29 17:01:38,640 DEBUG [c.c.s.StatsCollector]
> > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) HostStatsCollector is
> > running...
> > 2022-08-29 17:01:38,655 DEBUG [c.c.a.t.Request]
> > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
> 1-5410793477308947402:
> > Received: { Ans: , MgmtId: 90520740759984, via:
> > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> > 2022-08-29 17:01:38,707 DEBUG [c.c.a.t.Request]
> > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
> 5-8959067033723473395:
> > Received: { Ans: , MgmtId: 90520740759984, via:
> > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> > 2022-08-29 17:01:38,717 DEBUG [c.c.a.t.Request]
> > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
> 6-4833769775052102464:
> > Received: { Ans: , MgmtId: 90520740759984, via:
> > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> > 2022-08-29 17:01:38,725 DEBUG [c.c.a.t.Request]
> > (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq
> 7-295548725546192472:
> > Received: { Ans: , MgmtId: 90520740759984, via:
> > 7(srv-5-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> > 2022-08-29 17:01:41,709 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-5:ctx-a65ae462) (logid:bc2b7320) HA health
> check
> > task is running...
> > 2022-08-29 17:01:45,712 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-2:ctx-05491ace) (logid:b2232c4d) HA health
> check
> > task is running...
> > 2022-08-29 17:01:47,019 DEBUG [c.c.a.m.AgentManagerImpl]
> > (AgentManager-Handler-14:null) (logid:) Ping from Routing host
> > 7(srv-5-comp.aaaaaaaa.bbb)
> >
> > 2022-08-29 17:01:49,715 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-2:ctx-7943581c) (logid:504c98ff) HA health
> check
> > task is running...
> > 2022-08-29 17:01:51,881 DEBUG [c.c.a.m.AgentManagerImpl]
> > (AgentManager-Handler-13:null) (logid:) Ping from Routing host
> > 1(srv-2-comp.aaaaaaaa.bbb)
> >
> > 2022-08-29 17:01:53,718 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-4:ctx-b11cfa22) (logid:3163613f) HA health
> check
> > task is running...
> > 2022-08-29 17:01:53,955 DEBUG [c.c.a.t.Request]
> > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
> 1-5410793477308947403:
> > Received: { Ans: , MgmtId: 90520740759984, via:
> > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
> > 2022-08-29 17:01:54,025 DEBUG [c.c.a.t.Request]
> > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
> 5-8959067033723473396:
> > Received: { Ans: , MgmtId: 90520740759984, via:
> > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
> > 2022-08-29 17:01:54,096 DEBUG [c.c.a.t.Request]
> > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
> 6-4833769775052102465:
> > Received: { Ans: , MgmtId: 90520740759984, via:
> > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
> > 2022-08-29 17:01:54,217 DEBUG [c.c.a.t.Request]
> > (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq
> 7-295548725546192473:
> > Received: { Ans: , MgmtId: 90520740759984, via:
> > 7(srv-5-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
> > 2022-08-29 17:01:55,765 DEBUG [c.c.s.StatsCollector]
> > (secstorage-1:ctx-7d9995d7) (logid:e388a8b8) Verifying image storage [1].
> > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
> > 2022-08-29 17:01:55,771 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > (secstorage-1:ctx-7d9995d7) (logid:e388a8b8) Zone [1] is ready to launch
> > secondary storage VM.
> > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Zone 1 is ready to launch
> > console proxy
> > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Expand console proxy
> standby
> > capacity for zone xxxxxxxxxx
> > 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Allocating console proxy
> > standby capacity for zone [1].
> > 2022-08-29 17:01:56,034 INFO [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Found a stopped console
> > proxy, starting it. Vm id : 85
> > 2022-08-29 17:01:56,035 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) start parameter value of
> > enterHardwareSetup == <very null> during processing of queued job
> > 2022-08-29 17:01:56,044 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Sync job-4288 execution on
> > object VmWorkJobQueue.85
> > 2022-08-29 17:01:56,173 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-e75826b1) (logid:e80fe659) Execute
> sync-queue
> > item: SyncQueueItemVO {id:2105, queueId: 2105, contentType: AsyncJob,
> > contentId: 4288, lastProcessMsid: 90520740759984, lastprocessNumber: 1,
> > lastProcessTime: Mon Aug 29 17:01:56 CEST 2022, created: Mon Aug 29
> > 17:01:56 CEST 2022}
> > 2022-08-29 17:01:56,174 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-e75826b1) (logid:e80fe659) Schedule queued
> > job-4288
> > 2022-08-29 17:01:56,179 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:7e671dd0)
> Add
> > job-4288 into job monitoring
> > 2022-08-29 17:01:56,186 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5)
> > Executing AsyncJobVO: {id:4288, userId: 1, accountId: 1, instanceType:
> > null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
> >
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAVXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
> > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> > result: null, initMsid: 90520740759984, completeMsid: null, lastUpdated:
> > null, lastPolled: null, created: Mon Aug 29 17:01:56 CEST 2022, removed:
> > null}
> > 2022-08-29 17:01:56,186 DEBUG [c.c.v.VmWorkJobDispatcher]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5)
> Run
> > VM work job: com.cloud.vm.VmWorkStart for VM 85, job origin: 3445
> > 2022-08-29 17:01:56,187 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Execute VM work job:
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> > 2022-08-29 17:01:56,189 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) orchestrating VM start for 'v-85-VM'
> > com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
> > 2022-08-29 17:01:56,196 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
> > transited from [Stopped] to [Starting] with event [StartRequested]. VM's
> > original host: null, new host: null, host before state transition: null
> > 2022-08-29 17:01:56,196 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Successfully transitioned to start state for VM instance
> > {id: "85", name: "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
> > type="ConsoleProxy"} reservation id =
> 0ab68202-d8be-469c-9b2a-87a55c2dc489
> > 2022-08-29 17:01:56,200 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts: null
> > 2022-08-29 17:01:56,202 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) VM start attempt #1
> > 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
> > 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Trying to allocate a host and storage pools from dc:1,
> > pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
> > 1073741824
> > 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
> > 2022-08-29 17:01:56,212 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
> > 2022-08-29 17:01:56,214 DEBUG [c.c.d.FirstFitPlanner]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Searching all possible resources under this Zone: 1
> > 2022-08-29 17:01:56,215 DEBUG [c.c.d.FirstFitPlanner]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Listing clusters in order of aggregate capacity, that
> have
> > (at least one host with) enough CPU and RAM capacity under this Zone: 1
> > 2022-08-29 17:01:56,218 DEBUG [c.c.d.FirstFitPlanner]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Removing from the clusterId list these clusters from
> avoid
> > set: []
> > 2022-08-29 17:01:56,224 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
> > 2022-08-29 17:01:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested with
> > Custom[UEFI] Boot Type false
> > 2022-08-29 17:01:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in dc: 1
> > pod:1 cluster:1
> > 2022-08-29 17:01:56,226 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4 hosts
> to
> > check for allocation: [Host {"id": "5", "name":
> "srv-4-comp.aaaaaaaa.bbb",
> > "uuid": "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
> > {"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb", "uuid":
> > "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host {"id":
> "6",
> > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id":
> "1",
> > "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> > 2022-08-29 17:01:56,232 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for allocation
> > after prioritization: [Host {"id": "5", "name":
> "srv-4-comp.aaaaaaaa.bbb",
> > "uuid": "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
> > {"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb", "uuid":
> > "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host {"id":
> "6",
> > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id":
> "1",
> > "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> > 2022-08-29 17:01:56,232 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for speed=500Mhz,
> > Ram=1024 MB
> > 2022-08-29 17:01:56,232 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
> > srv-4-comp.aaaaaaaa.bbb, uuid: 134c625d-15ab-461a-a38d-36fb0b979b5d} is
> KVM
> > hypervisor type, no max guest limit check needed
> > 2022-08-29 17:01:56,234 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu capability
> > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> > 2022-08-29 17:01:56,234 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has enough
> > capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> > cpuOverprovisioningFactor: 1.0
> > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> > and CPU after applying overprovisioning: 127200
> > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 , Requested
> > CPU: 500
> > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
> > 251009040384 , Requested RAM: (1.00 GB) 1073741824
> > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > available
> > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> > 5, used: 11100, reserved: 0, actual total: 127200, total with
> > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > ,considerReservedCapacity?: true
> > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> > 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total: (250.27
> GB)
> > 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> > false , considerReservedCapacity?: true
> > 2022-08-29 17:01:56,235 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding
> to
> > list: 5
> > 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
> > srv-5-comp.aaaaaaaa.bbb, uuid: 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is
> KVM
> > hypervisor type, no max guest limit check needed
> > 2022-08-29 17:01:56,238 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu capability
> > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> > 2022-08-29 17:01:56,238 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has enough
> > capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> > cpuOverprovisioningFactor: 1.0
> > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> > and CPU after applying overprovisioning: 127200
> > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 , Requested
> CPU:
> > 500
> > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
> > 189373743104 , Requested RAM: (1.00 GB) 1073741824
> > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > available
> > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> > 7, used: 90100, reserved: 0, actual total: 127200, total with
> > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > ,considerReservedCapacity?: true
> > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> > 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total: (250.27
> GB)
> > 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> > false , considerReservedCapacity?: true
> > 2022-08-29 17:01:56,240 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding
> to
> > list: 7
> > 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
> > srv-3-comp.aaaaaaaa.bbb, uuid: 6858713b-7e17-46d8-854d-24a13f9c7001} is
> KVM
> > hypervisor type, no max guest limit check needed
> > 2022-08-29 17:01:56,242 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu capability
> > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> > 2022-08-29 17:01:56,242 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has enough
> > capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> > cpuOverprovisioningFactor: 1.0
> > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> > and CPU after applying overprovisioning: 127200
> > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 , Requested
> CPU:
> > 500
> > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
> > 204032835584 , Requested RAM: (1.00 GB) 1073741824
> > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > available
> > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> > 6, used: 42900, reserved: 0, actual total: 127200, total with
> > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > ,considerReservedCapacity?: true
> > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> > 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total: (250.27
> GB)
> > 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> > false , considerReservedCapacity?: true
> > 2022-08-29 17:01:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding
> to
> > list: 6
> > 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
> > srv-2-comp.aaaaaaaa.bbb, uuid: 883f7f62-55f2-42f4-8242-16b6847c4204} is
> KVM
> > hypervisor type, no max guest limit check needed
> > 2022-08-29 17:01:56,246 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu capability
> > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> > 2022-08-29 17:01:56,246 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has enough
> > capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> > cpuOverprovisioningFactor: 1.0
> > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> > and CPU after applying overprovisioning: 127200
> > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 , Requested
> CPU:
> > 500
> > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
> > 229802639360 , Requested RAM: (1.00 GB) 1073741824
> > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > available
> > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> > 1, used: 48200, reserved: 0, actual total: 127200, total with
> > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > ,considerReservedCapacity?: true
> > 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> > 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total: (250.27
> GB)
> > 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> > false , considerReservedCapacity?: true
> > 2022-08-29 17:01:56,247 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding
> to
> > list: 1
> > 2022-08-29 17:01:56,247 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator returning 4
> > suitable hosts
> > 2022-08-29 17:01:56,258 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
> > transited from [Starting] to [Stopped] with event [OperationFailed]. VM's
> > original host: null, new host: null, host before state transition: null
> > 2022-08-29 17:01:56,258 WARN [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Unable to orchestrate start VM instance {id: "85", name:
> > "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
> > type="ConsoleProxy"} due to [Unable to create deployment, no usable
> volumes
> > found for the VM: 85].
> > com.cloud.utils.exception.CloudRuntimeException: Unable to create
> > deployment, no usable volumes found for the VM: 85
> > at
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
> > at
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
> > at
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
> > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
> > at
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > at java.base/java.lang.Thread.run(Thread.java:829)
> > 2022-08-29 17:01:56,259 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Done executing VM work job:
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> > 2022-08-29 17:01:56,260 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Complete async job-4288, jobStatus: FAILED, resultCode:
> 0,
> > result:
> >
> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODUiLCBuYW1lOiAidi04NS1WTSIsIHV1aWQ6ICIxMTBkODYyMC1jMDI5LTRjYTktOTA4NC0yMTAwOTk3MjVhODAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODVdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
> > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Publish async job-4288 complete on message bus
> > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Wake up jobs related to job-4288
> > 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Update db status for job-4288
> > 2022-08-29 17:01:56,262 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> > (logid:ba10fec5) Wake up jobs joined with job-4288 and disjoin all
> subjobs
> > created from job- 4288
> > 2022-08-29 17:01:56,270 DEBUG [c.c.v.VmWorkJobDispatcher]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5)
> > Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 85, job
> > origin: 3445
> > 2022-08-29 17:01:56,270 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5)
> > Done executing com.cloud.vm.VmWorkStart for job-4288
> > 2022-08-29 17:01:56,272 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5)
> > Remove job-4288 from job monitoring
> > 2022-08-29 17:01:56,277 WARN [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Unable to allocate console
> > proxy standby capacity for zone [1] due to [Unable to orchestrate start
> VM
> > instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> [Unable
> > to create deployment, no usable volumes found for the VM: 85].].
> > com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate
> > start VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> [Unable
> > to create deployment, no usable volumes found for the VM: 85].
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
> > at
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > at java.base/java.lang.Thread.run(Thread.java:829)
> > 2022-08-29 17:01:56,277 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) received console proxy
> alert
> > 2022-08-29 17:01:56,278 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Console proxy creation
> > failure, zone: xxxxxxxxxx
> > 2022-08-29 17:01:56,279 WARN [c.c.a.AlertManagerImpl]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) alertType=[10]
> > dataCenterId=[1] podId=[null] clusterId=[null] message=[Console proxy
> > creation failure. zone: xxxxxxxxxx, error details: Unable to orchestrate
> > start VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> [Unable
> > to create deployment, no usable volumes found for the VM: 85].].
> > 2022-08-29 17:01:56,284 WARN [c.c.a.AlertManagerImpl]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) No recipients set in
> global
> > setting 'alert.email.addresses', skipping sending alert with subject
> > [Console proxy creation failure. zone: xxxxxxxxxx, error details: Unable
> to
> > orchestrate start VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> [Unable
> > to create deployment, no usable volumes found for the VM: 85].] and
> content
> > [Console proxy creation failure (zone xxxxxxxxxx)].
> > 2022-08-29 17:01:56,284 WARN [c.c.v.SystemVmLoadScanner]
> > (consoleproxy-1:ctx-413f712d) (logid:440fd101) Unexpected exception
> Unable
> > to orchestrate start VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> [Unable
> > to create deployment, no usable volumes found for the VM: 85].
> > com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate
> > start VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> [Unable
> > to create deployment, no usable volumes found for the VM: 85].
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
> > at
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > at java.base/java.lang.Thread.run(Thread.java:829)
> > 2022-08-29 17:01:57,721 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-4:ctx-9528226f) (logid:4512ff9b) HA health
> check
> > task is running...
> > 2022-08-29 17:01:59,504 DEBUG [c.c.s.StatsCollector]
> > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) StorageCollector is
> > running...
> > 2022-08-29 17:01:59,518 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
> getCommandHostDelegation:
> > class com.cloud.agent.api.GetStorageStatsCommand
> > 2022-08-29 17:01:59,518 DEBUG [c.c.h.XenServerGuru]
> > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning the
> > default host to execute commands because the command is not of Copy type.
> > 2022-08-29 17:01:59,571 DEBUG [c.c.a.t.Request]
> > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
> > 11-7523826127475836583: Received: { Ans: , MgmtId: 90520740759984, via:
> > 11(s-32-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> > 2022-08-29 17:01:59,574 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
> getCommandHostDelegation:
> > class com.cloud.agent.api.GetStorageStatsCommand
> > 2022-08-29 17:01:59,574 DEBUG [c.c.h.XenServerGuru]
> > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning the
> > default host to execute commands because the command is not of Copy type.
> > 2022-08-29 17:01:59,615 DEBUG [c.c.a.t.Request]
> > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
> 5-8959067033723473397:
> > Received: { Ans: , MgmtId: 90520740759984, via:
> > 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetStorageStatsAnswer
> } }
> > 2022-08-29 17:01:59,624 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8)
> getCommandHostDelegation:
> > class com.cloud.agent.api.GetStorageStatsCommand
> > 2022-08-29 17:01:59,624 DEBUG [c.c.h.XenServerGuru]
> > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning the
> > default host to execute commands because the command is not of Copy type.
> > 2022-08-29 17:01:59,653 DEBUG [c.c.a.t.Request]
> > (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
> 6-4833769775052102466:
> > Received: { Ans: , MgmtId: 90520740759984, via:
> > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetStorageStatsAnswer
> } }
> > 2022-08-29 17:01:59,979 DEBUG [c.c.a.m.AgentManagerImpl]
> > (AgentManager-Handler-12:null) (logid:) Ping from Routing host
> > 5(srv-4-comp.aaaaaaaa.bbb)
> >
> > 2022-08-29 17:02:00,269 DEBUG [c.c.s.StatsCollector]
> > (StatsCollector-5:ctx-d2b6ffd3) (logid:95201678) DbCollector is
> running...
> > 2022-08-29 17:02:00,443 DEBUG
> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 2 routers to
> > update status.
> > 2022-08-29 17:02:00,449 DEBUG
> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 1 VPC's to
> > update Redundant State.
> > 2022-08-29 17:02:00,450 DEBUG
> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 0 networks to
> > update RvR status.
> > 2022-08-29 17:02:00,457 DEBUG [c.c.a.t.Request]
> > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
> > 6-4833769775052102467: Sending { Cmd , MgmtId: 90520740759984, via:
> > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
> router.name
> ":"r-79-VM","router.ip":"169.254.144.198"},"wait":"30","bypassHostMaintenance":"false"}}]
> > }
> > 2022-08-29 17:02:00,599 DEBUG [c.c.a.t.Request]
> > (AgentManager-Handler-7:null) (logid:) Seq 6-4833769775052102467:
> > Processing: { Ans: , MgmtId: 90520740759984, via: 6, Ver: v1, Flags: 10,
> >
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"PRIMARY","result":"true","details":"Status:
> > PRIMARY
> > ","wait":"0","bypassHostMaintenance":"false"}}] }
> > 2022-08-29 17:02:00,599 DEBUG [c.c.a.t.Request]
> > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
> > 6-4833769775052102467: Received: { Ans: , MgmtId: 90520740759984, via:
> > 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer } }
> > 2022-08-29 17:02:00,599 DEBUG [c.c.a.m.AgentManagerImpl]
> > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Details
> > from executing class com.cloud.agent.api.CheckRouterCommand: Status:
> PRIMARY
> >
> > 2022-08-29 17:02:00,612 DEBUG [c.c.a.t.Request]
> > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
> > 1-5410793477308947404: Sending { Cmd , MgmtId: 90520740759984, via:
> > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> > [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"
> router.name
> ":"r-80-VM","router.ip":"169.254.217.200"},"wait":"30","bypassHostMaintenance":"false"}}]
> > }
> > 2022-08-29 17:02:00,748 DEBUG [c.c.a.m.AgentManagerImpl]
> > (AgentManager-Handler-4:null) (logid:) Ping from 11(s-32-VM)
> > 2022-08-29 17:02:00,750 DEBUG [c.c.a.t.Request]
> > (AgentManager-Handler-1:null) (logid:) Seq 1-5410793477308947404:
> > Processing: { Ans: , MgmtId: 90520740759984, via: 1, Ver: v1, Flags: 10,
> >
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"BACKUP","result":"true","details":"Status:
> > BACKUP
> > ","wait":"0","bypassHostMaintenance":"false"}}] }
> > 2022-08-29 17:02:00,750 DEBUG [c.c.a.t.Request]
> > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
> > 1-5410793477308947404: Received: { Ans: , MgmtId: 90520740759984, via:
> > 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer } }
> > 2022-08-29 17:02:00,751 DEBUG [c.c.a.m.AgentManagerImpl]
> > (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Details
> > from executing class com.cloud.agent.api.CheckRouterCommand: Status:
> BACKUP
> >
> > 2022-08-29 17:02:01,362 DEBUG [c.c.s.StatsCollector]
> > (Cluster-Worker-839:ctx-e097a8ab) (logid:3deacf65) StatusUpdate from
> > 90520740759984, json:
> >
> {"managementServerHostId":1,"managementServerHostUuid":"5615ca5f-0716-4da1-93ec-31f79fd19bbe","collectionTime":"Aug
> > 29, 2022, 5:02:01
> >
> PM","sessions":1,"cpuUtilization":0.0,"totalJvmMemoryBytes":324534272,"freeJvmMemoryBytes":68595424,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":103760388,"jvmStartTime":1661681560910,"availableProcessors":6,"loadAverage":0.02,"totalInit":139788288,"totalUsed":474791104,"totalCommitted":557662208,"pid":1522332,"jvmName":"1522332@vm-1-cs-mgmt.aaaaaaaa.bbb
> ","jvmVendor":"Ubuntu","jvmVersion":"11.0.16+8-post-Ubuntu-0ubuntu120.04","osDistribution":"Ubuntu
> > 20.04.5
> >
> LTS","agentCount":5,"heapMemoryUsed":256775312,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":13,"threadsRunnableCount":15,"threadsTerminatedCount":0,"threadsTotalCount":420,"threadsWaitingCount":331,"systemMemoryTotal":8343236608,"systemMemoryFree":2623193088,"systemMemoryUsed":814808,"systemMemoryVirtualSize":7494184960,"logInfo":"/var/log/cloudstack/management/management-server.log
> > using: 74M\non disk /dev/vda2 mounted on / (14%
> >
> full)","systemTotalCpuCycles":15870.180000000002,"systemLoadAverages":[0.02,0.03,0.0],"systemCyclesUsage":[4116569,2615101,2716460424],"dbLocal":false,"usageLocal":false,"systemBootTime":"Jul
> > 8, 2022, 2:35:55 AM","kernelVersion":"5.4.0-121-generic"}
> > 2022-08-29 17:02:01,723 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-4:ctx-7abb1b5d) (logid:2320950d) HA health
> check
> > task is running...
> > 2022-08-29 17:02:05,417 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (Vm-Operations-Cleanup-1:ctx-e741e208) (logid:3f5129e0) VM Operation
> Thread
> > Running
> > 2022-08-29 17:02:05,725 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-4:ctx-8e7f6766) (logid:710cb3b2) HA health
> check
> > task is running...
> > 2022-08-29 17:02:08,626 DEBUG [c.c.a.m.AgentManagerImpl]
> > (AgentManager-Handler-11:null) (logid:) Ping from Routing host
> > 6(srv-3-comp.aaaaaaaa.bbb)
> >
> > 2022-08-29 17:02:09,727 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-4:ctx-26a5ff07) (logid:09b3c89a) HA health
> check
> > task is running...
> > 2022-08-29 17:02:13,729 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-4:ctx-b4d4e374) (logid:9048c7a7) HA health
> check
> > task is running...
> > 2022-08-29 17:02:17,731 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-4:ctx-3db3d208) (logid:7bb73f7b) HA health
> check
> > task is running...
> > 2022-08-29 17:02:21,734 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-1:ctx-fca1d248) (logid:1ed5381a) HA health
> check
> > task is running...
> > 2022-08-29 17:02:25,737 DEBUG [o.a.c.h.HAManagerImpl]
> > (BackgroundTaskPollManager-1:ctx-984e2382) (logid:e6c3d724) HA health
> check
> > task is running...
> > 2022-08-29 17:02:25,759 DEBUG [c.c.s.StatsCollector]
> > (secstorage-1:ctx-64c610ed) (logid:6603cd38) Verifying image storage [1].
> > Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
> > 2022-08-29 17:02:25,763 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> > (secstorage-1:ctx-64c610ed) (logid:6603cd38) Zone [1] is ready to launch
> > secondary storage VM.
> > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Zone 1 is ready to launch
> > console proxy
> > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Expand console proxy
> standby
> > capacity for zone xxxxxxxxxx
> > 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Allocating console proxy
> > standby capacity for zone [1].
> > 2022-08-29 17:02:26,030 INFO [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Found a stopped console
> > proxy, starting it. Vm id : 85
> > 2022-08-29 17:02:26,031 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) start parameter value of
> > enterHardwareSetup == <very null> during processing of queued job
> > 2022-08-29 17:02:26,037 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Sync job-4289 execution on
> > object VmWorkJobQueue.85
> > 2022-08-29 17:02:26,167 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-bba6b89d) (logid:f9aed0eb) Execute
> sync-queue
> > item: SyncQueueItemVO {id:2106, queueId: 2105, contentType: AsyncJob,
> > contentId: 4289, lastProcessMsid: 90520740759984, lastprocessNumber: 2,
> > lastProcessTime: Mon Aug 29 17:02:26 CEST 2022, created: Mon Aug 29
> > 17:02:26 CEST 2022}
> > 2022-08-29 17:02:26,167 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (AsyncJobMgr-Heartbeat-1:ctx-bba6b89d) (logid:f9aed0eb) Schedule queued
> > job-4289
> > 2022-08-29 17:02:26,172 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:f919f9b3)
> Add
> > job-4289 into job monitoring
> > 2022-08-29 17:02:26,181 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5)
> > Executing AsyncJobVO: {id:4289, userId: 1, accountId: 1, instanceType:
> > null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
> >
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAVXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
> > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> > result: null, initMsid: 90520740759984, completeMsid: null, lastUpdated:
> > null, lastPolled: null, created: Mon Aug 29 17:02:26 CEST 2022, removed:
> > null}
> > 2022-08-29 17:02:26,181 DEBUG [c.c.v.VmWorkJobDispatcher]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5)
> Run
> > VM work job: com.cloud.vm.VmWorkStart for VM 85, job origin: 3445
> > 2022-08-29 17:02:26,183 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Execute VM work job:
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> > 2022-08-29 17:02:26,184 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) orchestrating VM start for 'v-85-VM'
> > com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
> > 2022-08-29 17:02:26,192 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
> > transited from [Stopped] to [Starting] with event [StartRequested]. VM's
> > original host: null, new host: null, host before state transition: null
> > 2022-08-29 17:02:26,192 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Successfully transitioned to start state for VM instance
> > {id: "85", name: "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
> > type="ConsoleProxy"} reservation id =
> 24bf955f-96d8-4f4f-ae33-2cae2f3e0425
> > 2022-08-29 17:02:26,198 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts: null
> > 2022-08-29 17:02:26,200 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) VM start attempt #1
> > 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
> > 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Trying to allocate a host and storage pools from dc:1,
> > pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
> > 1073741824
> > 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
> > 2022-08-29 17:02:26,211 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
> > 2022-08-29 17:02:26,212 DEBUG [c.c.d.FirstFitPlanner]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Searching all possible resources under this Zone: 1
> > 2022-08-29 17:02:26,213 DEBUG [c.c.d.FirstFitPlanner]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Listing clusters in order of aggregate capacity, that
> have
> > (at least one host with) enough CPU and RAM capacity under this Zone: 1
> > 2022-08-29 17:02:26,216 DEBUG [c.c.d.FirstFitPlanner]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Removing from the clusterId list these clusters from
> avoid
> > set: []
> > 2022-08-29 17:02:26,221 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
> > 2022-08-29 17:02:26,222 INFO [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested with
> > Custom[UEFI] Boot Type false
> > 2022-08-29 17:02:26,223 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in dc: 1
> > pod:1 cluster:1
> > 2022-08-29 17:02:26,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4 hosts
> to
> > check for allocation: [Host {"id": "7", "name":
> "srv-5-comp.aaaaaaaa.bbb",
> > "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
> > {"id": "1", "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}, Host {"id":
> "6",
> > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id":
> "5",
> > "name": "srv-4-comp.aaaaaaaa.bbb", "uuid":
> > "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}]
> > 2022-08-29 17:02:26,231 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for allocation
> > after prioritization: [Host {"id": "7", "name":
> "srv-5-comp.aaaaaaaa.bbb",
> > "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
> > {"id": "1", "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> > "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}, Host {"id":
> "6",
> > "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> > "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id":
> "5",
> > "name": "srv-4-comp.aaaaaaaa.bbb", "uuid":
> > "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}]
> > 2022-08-29 17:02:26,231 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Looking for speed=500Mhz,
> > Ram=1024 MB
> > 2022-08-29 17:02:26,231 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
> > srv-5-comp.aaaaaaaa.bbb, uuid: 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is
> KVM
> > hypervisor type, no max guest limit check needed
> > 2022-08-29 17:02:26,234 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu capability
> > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> > 2022-08-29 17:02:26,234 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has enough
> > capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> > cpuOverprovisioningFactor: 1.0
> > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> > and CPU after applying overprovisioning: 127200
> > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 , Requested
> CPU:
> > 500
> > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
> > 189373743104 , Requested RAM: (1.00 GB) 1073741824
> > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > available
> > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> > 7, used: 90100, reserved: 0, actual total: 127200, total with
> > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > ,considerReservedCapacity?: true
> > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> > 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total: (250.27
> GB)
> > 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> > false , considerReservedCapacity?: true
> > 2022-08-29 17:02:26,236 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding
> to
> > list: 7
> > 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
> > srv-2-comp.aaaaaaaa.bbb, uuid: 883f7f62-55f2-42f4-8242-16b6847c4204} is
> KVM
> > hypervisor type, no max guest limit check needed
> > 2022-08-29 17:02:26,239 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu capability
> > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> > 2022-08-29 17:02:26,239 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has enough
> > capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> > cpuOverprovisioningFactor: 1.0
> > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> > and CPU after applying overprovisioning: 127200
> > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 , Requested
> CPU:
> > 500
> > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
> > 229802639360 , Requested RAM: (1.00 GB) 1073741824
> > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > available
> > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> > 1, used: 48200, reserved: 0, actual total: 127200, total with
> > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > ,considerReservedCapacity?: true
> > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> > 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total: (250.27
> GB)
> > 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> > false , considerReservedCapacity?: true
> > 2022-08-29 17:02:26,240 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding
> to
> > list: 1
> > 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
> > srv-3-comp.aaaaaaaa.bbb, uuid: 6858713b-7e17-46d8-854d-24a13f9c7001} is
> KVM
> > hypervisor type, no max guest limit check needed
> > 2022-08-29 17:02:26,243 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu capability
> > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> > 2022-08-29 17:02:26,243 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has enough
> > capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> > cpuOverprovisioningFactor: 1.0
> > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> > and CPU after applying overprovisioning: 127200
> > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 , Requested
> CPU:
> > 500
> > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
> > 204032835584 , Requested RAM: (1.00 GB) 1073741824
> > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > available
> > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> > 6, used: 42900, reserved: 0, actual total: 127200, total with
> > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > ,considerReservedCapacity?: true
> > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> > 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total: (250.27
> GB)
> > 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> > false , considerReservedCapacity?: true
> > 2022-08-29 17:02:26,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding
> to
> > list: 6
> > 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
> > srv-4-comp.aaaaaaaa.bbb, uuid: 134c625d-15ab-461a-a38d-36fb0b979b5d} is
> KVM
> > hypervisor type, no max guest limit check needed
> > 2022-08-29 17:02:26,247 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu capability
> > (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> > 2022-08-29 17:02:26,247 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has enough
> > capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> > cpuOverprovisioningFactor: 1.0
> > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> > and CPU after applying overprovisioning: 127200
> > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 , Requested
> > CPU: 500
> > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
> > 251009040384 , Requested RAM: (1.00 GB) 1073741824
> > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> > available
> > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> > 5, used: 11100, reserved: 0, actual total: 127200, total with
> > overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> > ,considerReservedCapacity?: true
> > 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> > 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total: (250.27
> GB)
> > 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> > false , considerReservedCapacity?: true
> > 2022-08-29 17:02:26,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding
> to
> > list: 5
> > 2022-08-29 17:02:26,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> > FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator returning 4
> > suitable hosts
> > 2022-08-29 17:02:26,260 DEBUG [c.c.c.CapacityManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
> > transited from [Starting] to [Stopped] with event [OperationFailed]. VM's
> > original host: null, new host: null, host before state transition: null
> > 2022-08-29 17:02:26,260 WARN [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Unable to orchestrate start VM instance {id: "85", name:
> > "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
> > type="ConsoleProxy"} due to [Unable to create deployment, no usable
> volumes
> > found for the VM: 85].
> > com.cloud.utils.exception.CloudRuntimeException: Unable to create
> > deployment, no usable volumes found for the VM: 85
> > at
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
> > at
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
> > at
> >
> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
> > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
> > at
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > at java.base/java.lang.Thread.run(Thread.java:829)
> > 2022-08-29 17:02:26,261 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Done executing VM work job:
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> > 2022-08-29 17:02:26,262 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Complete async job-4289, jobStatus: FAILED, resultCode:
> 0,
> > result:
> >
> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODUiLCBuYW1lOiAidi04NS1WTSIsIHV1aWQ6ICIxMTBkODYyMC1jMDI5LTRjYTktOTA4NC0yMTAwOTk3MjVhODAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODVdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
> > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Publish async job-4289 complete on message bus
> > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Wake up jobs related to job-4289
> > 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Update db status for job-4289
> > 2022-08-29 17:02:26,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> > (logid:ba10fec5) Wake up jobs joined with job-4289 and disjoin all
> subjobs
> > created from job- 4289
> > 2022-08-29 17:02:26,269 DEBUG [c.c.v.VmWorkJobDispatcher]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5)
> > Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 85, job
> > origin: 3445
> > 2022-08-29 17:02:26,270 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5)
> > Done executing com.cloud.vm.VmWorkStart for job-4289
> > 2022-08-29 17:02:26,271 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5)
> > Remove job-4289 from job monitoring
> > 2022-08-29 17:02:26,277 WARN [c.c.c.ConsoleProxyManagerImpl]
> > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Unable to allocate console
> > proxy standby capacity for zone [1] due to [Unable to orchestrate start
> VM
> > instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> [Unable
> > to create deployment, no usable volumes found for the VM: 85].].
> > com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate
> > start VM instance {id: "85", name: "v-85-VM", uuid:
> > "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to
> [Unable
> > to create deployment, no usable volumes found for the VM: 85].
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> > at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
> > at
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> > at
> >
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> > at
> >
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> > at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> > at
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> > at java.base/java.lang.Thread.run(Thread.java:829)
> > 2022-08-29 17:02:26,277 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> > (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) received console proxy
> alert
> >
> >
> >
> > Am Mo., 29. Aug. 2022 um 16:30 Uhr schrieb Slavka Peleva <
> > slavkap@storpool.com>:
> >
> >> Thanks, Chris, for the information!
> >>
> >> Are you using custom offerings for the system VMs? Can you share more
> >> info if yes?
> >>
> >> About the log, it's not the complete log from where the deployment
> >> started. The deployment begins a bit earlier than what you shared. Can
> you
> >> search the management log for `Sync job-3539 execution on object
> >> VmWorkJobQueue` to get the `logid` for the volume creation (coloured in
> >> blue)? And if possible, share the logs with the found log ID
> >>
> >> Here is an example from my logs:
> >>
> >> 2022-08-29 17:07:49,114 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (consoleproxy-1:ctx-a952a3dc) (logid:89157bb9) Sync job-60484 execution
> >> on object VmWorkJobQueue.4904
> >>
> >> 2022-08-29 17:07:50,816 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> >> (Work-Job-Executor-5:ctx-56172581 job-60479/job-60484 ctx-a900986d)
> >> (logid:95d750ae) Execute VM work job:
> >>
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":4904,"handlerName":"VirtualMachineManagerImpl"}
> >> 2022-08-29 17:07:50,817 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >> (Work-Job-Executor-5:ctx-56172581 job-60479/job-60484 ctx-a900986d)
> >> (logid:95d750ae) orchestrating VM start for 'v-4904-VM'
> >> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
> >>
> >>
> >> On Mon, Aug 29, 2022 at 2:29 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
> >>
> >>> Hi Slavka,
> >>>
> >>> thats what i did after your suggestion / informatin and where the error
> >>> occured.
> >>> Deleted / Destroyed the consoleproxy and waited till recreation. Which
> >>> didn't worked throwing the error i pasted above.
> >>> Sorry for beeing so unspecific.
> >>>
> >>> Regards,
> >>> Chris
> >>>
> >>> Am Mo., 29. Aug. 2022 um 12:46 Uhr schrieb Slavka Peleva <
> >>> slavkap@storpool.com>:
> >>>
> >>>> Hi Chris,
> >>>>
> >>>> I mean to recreate the system VMs by destroying them.
> >>>>
> >>>> On Mon, Aug 29, 2022 at 12:38 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
> >>>>
> >>>>> Hi Slavka,
> >>>>>
> >>>>> didn't tried to restart the SystemVMs. Reading through the docs i had
> >>>>> the impression that "live-patching" would be enough for this....
> >>>>> Anyway - I tried my luck with the console proxy.
> >>>>>
> >>>>> Which failed utterly. Not starting as it isn't finding any useable
> >>>>> storage pool. - logs at the end of the mail.
> >>>>>
> >>>>> I tried deploying a VM using the same storage pool which worked
> >>>>> wonderfully.
> >>>>> There i noticed that all my custome storage-offerings are unuseable
> at
> >>>>> the moment. They are not accepting any custome storage sizes
> anymore...
> >>>>> Throwing failures "VM Creation failed. Volume size: -1GB is out of
> allowed
> >>>>> tange. Max:3072 Min:1"...
> >>>>> So this is also currently broken....
> >>>>>
> >>>>> 2022-08-29 10:51:56,189 DEBUG [c.c.v.VmWorkJobDispatcher]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
> (logid:ba10fec5) Run
> >>>>> VM work job: com.cloud.vm.VmWorkStart for VM 83, job origin: 3445
> >>>>> 2022-08-29 10:51:56,191 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Execute VM work job:
> >>>>>
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
> >>>>> 2022-08-29 10:51:56,192 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) orchestrating VM start for 'v-83-VM'
> >>>>> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
> >>>>> 2022-08-29 10:51:56,198 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
> >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} state
> >>>>> transited from [Stopped] to [Starting] with event [StartRequested].
> VM's
> >>>>> original host: null, new host: null, host before state transition:
> null
> >>>>> 2022-08-29 10:51:56,198 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Successfully transitioned to start state for VM
> instance
> >>>>> {id: "83", name: "v-83-VM", uuid:
> "b1d1f9c6-eafe-4527-9335-e649c646aab0",
> >>>>> type="ConsoleProxy"} reservation id =
> 6641ccc0-bcef-4b1a-a1f7-8dd8000626ad
> >>>>> 2022-08-29 10:51:56,203 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts:
> null
> >>>>> 2022-08-29 10:51:56,206 DEBUG [c.c.v.VirtualMachineManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) VM start attempt #1
> >>>>> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
> >>>>> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Trying to allocate a host and storage pools from
> dc:1,
> >>>>> pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
> >>>>> 1073741824
> >>>>> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
> >>>>> 2022-08-29 10:51:56,214 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
> >>>>> 2022-08-29 10:51:56,215 DEBUG [c.c.d.FirstFitPlanner]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Searching all possible resources under this Zone: 1
> >>>>> 2022-08-29 10:51:56,216 DEBUG [c.c.d.FirstFitPlanner]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Listing clusters in order of aggregate capacity,
> that have
> >>>>> (at least one host with) enough CPU and RAM capacity under this
> Zone: 1
> >>>>> 2022-08-29 10:51:56,219 DEBUG [c.c.d.FirstFitPlanner]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Removing from the clusterId list these clusters
> from avoid
> >>>>> set: []
> >>>>> 2022-08-29 10:51:56,224 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
> >>>>> 2022-08-29 10:51:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested
> with
> >>>>> Custom[UEFI] Boot Type false
> >>>>> 2022-08-29 10:51:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in dc: 1
> >>>>> pod:1 cluster:1
> >>>>> 2022-08-29 10:51:56,227 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4
> hosts to
> >>>>> check for allocation: [Host {"id": "7", "name":
> "srv-5-comp.blabla.blub",
> >>>>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"},
> Host
> >>>>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
> >>>>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
> {"id": "6",
> >>>>> "name": "srv-3-comp.blabla.blub", "uuid":
> >>>>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
> {"id": "1",
> >>>>> "name": "srv-2-comp.blabla.blub", "uuid":
> >>>>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for
> allocation
> >>>>> after prioritization: [Host {"id": "7", "name":
> "srv-5-comp.blabla.blub",
> >>>>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"},
> Host
> >>>>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
> >>>>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
> {"id": "6",
> >>>>> "name": "srv-3-comp.blabla.blub", "uuid":
> >>>>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host
> {"id": "1",
> >>>>> "name": "srv-2-comp.blabla.blub", "uuid":
> >>>>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for speed=500Mhz,
> >>>>> Ram=1024 MB
> >>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
> >>>>> srv-5-comp.blabla.blub, uuid: 8c753ce1-0253-49c9-9c7f-1f971a719ce6}
> is KVM
> >>>>> hypervisor type, no max guest limit check needed
> >>>>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu capability
> >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
> speed: 500
> >>>>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has
> enough
> >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> 1073741824 ,
> >>>>> cpuOverprovisioningFactor: 1.0
> >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> >>>>> and CPU after applying overprovisioning: 127200
> >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 ,
> Requested CPU:
> >>>>> 500
> >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
> >>>>> 189373743104 , Requested RAM: (1.00 GB) 1073741824
> >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
> RAM
> >>>>> available
> >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> >>>>> 7, used: 90100, reserved: 0, actual total: 127200, total with
> >>>>> overprovisioning: 127200; requested
> cpu:500,alloc_from_last_host?:false
> >>>>> ,considerReservedCapacity?: true
> >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> >>>>> 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total:
> (250.27 GB)
> >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
> alloc_from_last_host?:
> >>>>> false , considerReservedCapacity?: true
> >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> adding to
> >>>>> list: 7
> >>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
> >>>>> srv-4-comp.blabla.blub, uuid: 134c625d-15ab-461a-a38d-36fb0b979b5d}
> is KVM
> >>>>> hypervisor type, no max guest limit check needed
> >>>>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu capability
> >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
> speed: 500
> >>>>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has
> enough
> >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> 1073741824 ,
> >>>>> cpuOverprovisioningFactor: 1.0
> >>>>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> >>>>> and CPU after applying overprovisioning: 127200
> >>>>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 ,
> Requested
> >>>>> CPU: 500
> >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
> >>>>> 251009040384 , Requested RAM: (1.00 GB) 1073741824
> >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
> RAM
> >>>>> available
> >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> >>>>> 5, used: 11100, reserved: 0, actual total: 127200, total with
> >>>>> overprovisioning: 127200; requested
> cpu:500,alloc_from_last_host?:false
> >>>>> ,considerReservedCapacity?: true
> >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> >>>>> 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total:
> (250.27 GB)
> >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
> alloc_from_last_host?:
> >>>>> false , considerReservedCapacity?: true
> >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> adding to
> >>>>> list: 5
> >>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
> >>>>> srv-3-comp.blabla.blub, uuid: 6858713b-7e17-46d8-854d-24a13f9c7001}
> is KVM
> >>>>> hypervisor type, no max guest limit check needed
> >>>>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu capability
> >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
> speed: 500
> >>>>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has
> enough
> >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> 1073741824 ,
> >>>>> cpuOverprovisioningFactor: 1.0
> >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> >>>>> and CPU after applying overprovisioning: 127200
> >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 ,
> Requested CPU:
> >>>>> 500
> >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
> >>>>> 204032835584 , Requested RAM: (1.00 GB) 1073741824
> >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
> RAM
> >>>>> available
> >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> >>>>> 6, used: 42900, reserved: 0, actual total: 127200, total with
> >>>>> overprovisioning: 127200; requested
> cpu:500,alloc_from_last_host?:false
> >>>>> ,considerReservedCapacity?: true
> >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> >>>>> 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total:
> (250.27 GB)
> >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
> alloc_from_last_host?:
> >>>>> false , considerReservedCapacity?: true
> >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> adding to
> >>>>> list: 6
> >>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
> >>>>> srv-2-comp.blabla.blub, uuid: 883f7f62-55f2-42f4-8242-16b6847c4204}
> is KVM
> >>>>> hypervisor type, no max guest limit check needed
> >>>>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu capability
> >>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested
> speed: 500
> >>>>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has
> enough
> >>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB)
> 1073741824 ,
> >>>>> cpuOverprovisioningFactor: 1.0
> >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU:
> 127200
> >>>>> and CPU after applying overprovisioning: 127200
> >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 ,
> Requested CPU:
> >>>>> 500
> >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
> >>>>> 229802639360 , Requested RAM: (1.00 GB) 1073741824
> >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and
> RAM
> >>>>> available
> >>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from
> host:
> >>>>> 1, used: 48200, reserved: 0, actual total: 127200, total with
> >>>>> overprovisioning: 127200; requested
> cpu:500,alloc_from_last_host?:false
> >>>>> ,considerReservedCapacity?: true
> >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from
> host:
> >>>>> 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total:
> (250.27 GB)
> >>>>> 268725780480; requested mem: (1.00 GB) 1073741824,
> alloc_from_last_host?:
> >>>>> false , considerReservedCapacity?: true
> >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host,
> adding to
> >>>>> list: 1
> >>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> >>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator returning 4
> >>>>> suitable hosts
> >>>>> 2022-08-29 10:51:56,260 DEBUG [c.c.c.CapacityManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
> >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} state
> >>>>> transited from [Starting] to [Stopped] with event [OperationFailed].
> VM's
> >>>>> original host: null, new host: null, host before state transition:
> null
> >>>>> 2022-08-29 10:51:56,260 WARN [c.c.v.VirtualMachineManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Unable to orchestrate start VM instance {id: "83",
> name:
> >>>>> "v-83-VM", uuid: "b1d1f9c6-eafe-4527-9335-e649c646aab0",
> >>>>> type="ConsoleProxy"} due to [Unable to create deployment, no usable
> volumes
> >>>>> found for the VM: 83].
> >>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to create
> >>>>> deployment, no usable volumes found for the VM: 83
> >>>>> at
> >>>>>
> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
> >>>>> at
> >>>>>
> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
> >>>>> at
> >>>>>
> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
> >>>>> at
> >>>>>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
> >>>>> at
> >>>>>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
> >>>>> at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> >>>>> Source)
> >>>>> at
> >>>>>
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >>>>> at
> >>>>>
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> >>>>> at
> >>>>>
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> >>>>> at
> >>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> >>>>> at
> >>>>>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> >>>>> at
> >>>>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> >>>>> at
> >>>>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> >>>>> at
> >>>>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> >>>>> at
> >>>>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> >>>>> at
> >>>>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> >>>>> at
> >>>>>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> >>>>> at
> >>>>>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> >>>>> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >>>>> at
> >>>>>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >>>>> at
> >>>>>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >>>>> at java.base/java.lang.Thread.run(Thread.java:829)
> >>>>> 2022-08-29 10:51:56,262 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Done executing VM work job:
> >>>>>
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
> >>>>> 2022-08-29 10:51:56,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Complete async job-3539, jobStatus: FAILED,
> resultCode: 0,
> >>>>> result:
> >>>>>
> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODMiLCBuYW1lOiAidi04My1WTSIsIHV1aWQ6ICJiMWQxZjljNi1lYWZlLTQ1MjctOTMzNS1lNjQ5YzY0NmFhYjAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODNdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
> >>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Publish async job-3539 complete on message bus
> >>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Wake up jobs related to job-3539
> >>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Update db status for job-3539
> >>>>> 2022-08-29 10:51:56,266 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> >>>>> (logid:ba10fec5) Wake up jobs joined with job-3539 and disjoin all
> subjobs
> >>>>> created from job- 3539
> >>>>> 2022-08-29 10:51:56,276 DEBUG [c.c.v.VmWorkJobDispatcher]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
> (logid:ba10fec5) Done
> >>>>> with run of VM work job: com.cloud.vm.VmWorkStart for VM 83, job
> origin:
> >>>>> 3445
> >>>>> 2022-08-29 10:51:56,276 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
> (logid:ba10fec5) Done
> >>>>> executing com.cloud.vm.VmWorkStart for job-3539
> >>>>> 2022-08-29 10:51:56,278 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539)
> (logid:ba10fec5)
> >>>>> Remove job-3539 from job monitoring
> >>>>> 2022-08-29 10:51:56,285 WARN [c.c.c.ConsoleProxyManagerImpl]
> >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Unable to allocate
> console
> >>>>> proxy standby capacity for zone [1] due to [Unable to orchestrate
> start VM
> >>>>> instance {id: "83", name: "v-83-VM", uuid:
> >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to
> [Unable
> >>>>> to create deployment, no usable volumes found for the VM: 83].].
> >>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to
> orchestrate
> >>>>> start VM instance {id: "83", name: "v-83-VM", uuid:
> >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to
> [Unable
> >>>>> to create deployment, no usable volumes found for the VM: 83].
> >>>>> at
> >>>>>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> >>>>> at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
> >>>>> Source)
> >>>>> at
> >>>>>
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >>>>> at
> >>>>>
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> >>>>> at
> >>>>>
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> >>>>> at
> >>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> >>>>> at
> >>>>>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> >>>>> at
> >>>>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> >>>>> at
> >>>>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> >>>>> at
> >>>>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> >>>>> at
> >>>>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> >>>>> at
> >>>>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> >>>>> at
> >>>>>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> >>>>> at
> >>>>>
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> >>>>> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >>>>> at
> >>>>>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >>>>> at
> >>>>>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >>>>> at java.base/java.lang.Thread.run(Thread.java:829)
> >>>>> 2022-08-29 10:51:56,287 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) received console
> proxy alert
> >>>>> 2022-08-29 10:51:56,288 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Console proxy creation
> >>>>> failure, zone: blabla
> >>>>> 2022-08-29 10:51:56,289 WARN [c.c.a.AlertManagerImpl]
> >>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) alertType=[10]
> >>>>> dataCenterId=[1] podId=[null] clusterId=[null] message=[Console proxy
> >>>>> creation failure. zone: blabla, error details: Unable to orchestrate
> start
> >>>>> VM instance {id: "83", name: "v-83-VM", uuid:
> >>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to
> [Unable
> >>>>> to create deployment, no usable volumes found for the VM: 83].].
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> Am Mo., 29. Aug. 2022 um 09:40 Uhr schrieb Slavka Peleva
> >>>>> <sl...@storpool.com.invalid>:
> >>>>>
> >>>>>> Hi Chris,
> >>>>>>
> >>>>>> Did you recreate the system VMs? In the 4.17 version, the
> >>>>>> `systemvm.iso` is
> >>>>>> deprecated. In its place, you should have the file on agents
> >>>>>> `/usr/share/cloudstack-common/vms/agent.zip`.
> >>>>>> Can you share the complete log if the system VMs are newly created
> >>>>>> after
> >>>>>> the upgrade?
> >>>>>>
> >>>>>> Best regards,
> >>>>>> Slavka
> >>>>>>
> >>>>>> On Sun, Aug 28, 2022 at 2:16 PM Vash_X@gmx.de <Va...@gmx.de>
> wrote:
> >>>>>>
> >>>>>> > Hi everyone,
> >>>>>> >
> >>>>>> > faceing some challanges again after upgradeing to 4.17.
> >>>>>> > Did as explained in the docs.
> >>>>>> >
> >>>>>> > Afterwards I am now not able to migrate system VMs to different
> >>>>>> hosts.
> >>>>>> > Getting the error
> >>>>>> > "Exception during migrate: org.libvirt.LibvirtException: Cannot
> >>>>>> access
> >>>>>> > storage file '/usr/share/cloudstack-common/vms/systemvm.iso': No
> >>>>>> such file
> >>>>>> > or directory"
> >>>>>> >
> >>>>>> > Checked the exitstence of the file on the servers:
> >>>>>> > Mangement-Server has a "systemvm.iso.bak"
> >>>>>> > Hosts - no "systemvm.iso" at all present.
> >>>>>> >
> >>>>>> > Maybe someone has an idea on how to move on with this?
> >>>>>> >
> >>>>>> > Regards,
> >>>>>> > Chris
> >>>>>> >
> >>>>>>
> >>>>>
>
Re: Error migrate System VMs after upgrageing to 4.17
Posted by Slavka Peleva <sl...@storpool.com.INVALID>.
Hi Chris,
The deployment fails because CS cannot find a free IP address to allocate
for the system VM. Can you check if you explicitly added a range for the
system VMs, or you can check with this `select * from user_ip_address where
forsystemvms = 1;` If you don't have any ranges, you can get free IPs
with `SELECT
* FROM cloud.user_ip_address where allocated IS NULL AND removed IS NULL;`
. To get the exact query executed by CS, you need to increase the log level
to TRACE.
Best regards,
Slavka
On Mon, Aug 29, 2022 at 8:03 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
> Hi Slavka,
>
> no, i am not using any kind of custome offerings.
>
> I try to provide a more detailled log this time.
> Situation:
> v-83-VM was the "broken" consoleproxy. So the following log starts with
> it's expunging.
> Afterwards the new consoleproxy shall be deployed - v-85-VM.
>
> This first deployment (logid:dcfe0999) fails due to Insufficent address
> capacity - which i can't really understand or see in the database, checking
> the availeable IPs for systemVMs (given that i might looking up the wrong
> tables....).
> How ever CS is working as expected trying to deploy the console proxy
> (logid:440fd101) Sync job-4288).
> That job fails with "Unable to create deployment, no usable volumes found
> for the VM: 85".
> Afterwards i have another try to start the consoleproxy (job-4289) which
> fails for the same reason. And thats how things go on in the log file...
>
>
> 2022-08-29 17:01:16,224 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> Cleaning up hypervisor data structures (ex. SRs in XenServer) for managed
> storage. Data from VM instance {id: "83", name: "v-83-VM", uuid:
> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}.
> 2022-08-29 17:01:16,225 DEBUG [o.a.c.e.o.VolumeOrchestrator]
> (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> Cleaning storage for vm: 83
> 2022-08-29 17:01:16,233 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> Expunged VM instance {id: "83", name: "v-83-VM", uuid:
> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
> 2022-08-29 17:01:16,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> Complete async job-4286, jobStatus: SUCCEEDED, resultCode: 0, result:
> org.apache.cloudstack.api.response.SystemVmResponse/systemvm/{"id":"b1d1f9c6-eafe-4527-9335-e649c646aab0","systemvmtype":"consoleproxy","zoneid":"48e2e928-3300-43b5-8e3a-d9b358022502","zonename":"xxxxxxxxxx","dns1":"10.129.0.4","dns2":"195.10.195.195","name":"v-83-VM","templateid":"c20799e4-a990-449a-aae6-4b67374c7aff","templatename":"systemvm-kvm-4.17.0","created":"2022-08-29T10:38:26+0200","state":"Stopped","isdynamicallyscalable":"false","hasannotations":"false"}
> 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> Publish async job-4286 complete on message bus
> 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> Wake up jobs related to job-4286
> 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> Update db status for job-4286
> 2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
> Wake up jobs joined with job-4286 and disjoin all subjobs created from job-
> 4286
> 2022-08-29 17:01:16,260 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-35:ctx-29e06f28 job-4286) (logid:5fe8559a) Done executing
> org.apache.cloudstack.api.command.admin.systemvm.DestroySystemVmCmd for
> job-4286
> 2022-08-29 17:01:16,260 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-35:ctx-29e06f28 job-4286) (logid:5fe8559a) Remove
> job-4286 from job monitoring
> 2022-08-29 17:01:17,691 DEBUG [o.a.c.h.HAManagerImpl]
> (BackgroundTaskPollManager-6:ctx-6af18142) (logid:fe263a70) HA health check
> task is running...
>
> 2022-08-29 17:01:21,694 DEBUG [o.a.c.h.HAManagerImpl]
> (BackgroundTaskPollManager-6:ctx-89b1442f) (logid:0efc40b2) HA health check
> task is running...
> 2022-08-29 17:01:25,698 DEBUG [o.a.c.h.HAManagerImpl]
> (BackgroundTaskPollManager-5:ctx-62313d05) (logid:7073663f) HA health check
> task is running...
> 2022-08-29 17:01:25,767 DEBUG [c.c.s.StatsCollector]
> (secstorage-1:ctx-c80e5a6e) (logid:f8e3eb60) Verifying image storage [1].
> Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
> 2022-08-29 17:01:25,773 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-c80e5a6e) (logid:f8e3eb60) Zone [1] is ready to launch
> secondary storage VM.
> 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Zone 1 is ready to launch
> console proxy
> 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Expand console proxy standby
> capacity for zone xxxxxxxxxx
> 2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating console proxy
> standby capacity for zone [1].
> 2022-08-29 17:01:26,032 INFO [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) No stopped console proxy is
> available, need to allocate a new console proxy
> 2022-08-29 17:01:26,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Assign console proxy from a
> newly started instance for request from data center : 1
> 2022-08-29 17:01:26,056 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing network
> configuration for offering [Network Offering
> [1-Public-System-Public-Network]: Network {"id": 200, "name": "null",
> "uuid": "1b0bf965-a391-478d-833d-3e37e4a15eb8", "networkofferingid": 1}
> 2022-08-29 17:01:26,056 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
> Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account {"id": 1,
> "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
> 2022-08-29 17:01:26,064 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing network
> configuration for offering [Network Offering
> [3-Control-System-Control-Network]: Network {"id": 202, "name": "null",
> "uuid": "2af42518-fbf4-4a35-9d6b-370e6ad315c7", "networkofferingid": 3}
> 2022-08-29 17:01:26,064 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
> Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account {"id": 1,
> "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
> 2022-08-29 17:01:26,069 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing network
> configuration for offering [Network Offering
> [2-Management-System-Management-Network]: Network {"id": 201, "name":
> "null", "uuid": "53711d87-a3c6-4f33-bfce-c30e0de72760",
> "networkofferingid": 2}
> 2022-08-29 17:01:26,069 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
> Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account {"id": 1,
> "name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
> 2022-08-29 17:01:26,077 INFO [c.c.v.VirtualMachineManagerImpl]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) allocating virtual machine
> from template:c20799e4-a990-449a-aae6-4b67374c7aff with hostname:v-85-VM
> and 3 networks
> 2022-08-29 17:01:26,079 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating entries for VM:
> VM instance {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
> 2022-08-29 17:01:26,082 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating nics for VM
> instance {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
> 2022-08-29 17:01:26,083 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating nic for vm VM
> instance {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} in network
> Network {"id": 200, "name": "null", "uuid":
> "1b0bf965-a391-478d-833d-3e37e4a15eb8", "networkofferingid": 1} with
> requested profile NicProfile
> {"broadcastUri":null,"iPv4Address":null,"id":0,"reservationId":null,"vmId":0}
> 2022-08-29 17:01:26,089 WARN [c.c.n.IpAddressManagerImpl]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to get ip address in
> zone id=1, vlanId id=[3, 10, 13, 14]
> 2022-08-29 17:01:26,090 DEBUG [c.c.u.d.T.Transaction]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Rolling back the
> transaction: Time = 8 Name = consoleproxy-1; called by
> -TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-Transaction.execute:38-IpAddressManagerImpl.listAvailablePublicIps:824-IpAddressManagerImpl.fetchNewPublicIp:811-IpAddressManagerImpl.fetchNewPublicIp:803-IpAddressManagerImpl.assignPublicIpAddress:784-PublicNetworkGuru.getIp:125-PublicNetworkGuru.allocate:172-NetworkOrchestrator.allocateNic:973-NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice:823
> 2022-08-29 17:01:26,090 WARN [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to allocate proxy
> [Console VM instance {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
> due to [Insufficient address capacity].
> com.cloud.exception.InsufficientAddressCapacityException: Insufficient
> address capacityScope=interface com.cloud.dc.DataCenter; id=1
> at
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
> at
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> at
> com.cloud.network.IpAddressManagerImpl.listAvailablePublicIps(IpAddressManagerImpl.java:824)
> at
> com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl.java:811)
> at
> com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl.java:803)
> at
> com.cloud.network.IpAddressManagerImpl.assignPublicIpAddress(IpAddressManagerImpl.java:784)
> at
> com.cloud.network.guru.PublicNetworkGuru.getIp(PublicNetworkGuru.java:125)
> at
> com.cloud.network.guru.PublicNetworkGuru.allocate(PublicNetworkGuru.java:172)
> at
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocateNic(NetworkOrchestrator.java:973)
> at
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice(NetworkOrchestrator.java:823)
> at
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$3.doInTransactionWithoutResult(NetworkOrchestrator.java:797)
> at
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
> at
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> at
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocate(NetworkOrchestrator.java:778)
> at
> com.cloud.vm.VirtualMachineManagerImpl$1.doInTransactionWithoutResult(VirtualMachineManagerImpl.java:478)
> at
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
> at
> com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> at
> com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:469)
> at
> com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:528)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:700)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
> at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
> at
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
> at
> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> 2022-08-29 17:01:26,091 WARN [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to allocate console
> proxy standby capacity for zone [1] due to [Unable to allocate proxy
> [Console VM instance {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
> due to [Insufficient address capacity].].
> com.cloud.utils.exception.CloudRuntimeException: Unable to allocate proxy
> [Console VM instance {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
> due to [Insufficient address capacity].
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:704)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
> at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
> at
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
> at
> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: com.cloud.exception.InsufficientAddressCapacityException:
> Insufficient address capacityScope=interface com.cloud.dc.DataCenter; id=1
> at
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
> at
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> ... 38 more
> 2022-08-29 17:01:26,091 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) received console proxy alert
> 2022-08-29 17:01:26,092 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Console proxy creation
> failure, zone: xxxxxxxxxx
> 2022-08-29 17:01:26,093 WARN [c.c.a.AlertManagerImpl]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) alertType=[10]
> dataCenterId=[1] podId=[null] clusterId=[null] message=[Console proxy
> creation failure. zone: xxxxxxxxxx, error details: Unable to allocate proxy
> [Console VM instance {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
> due to [Insufficient address capacity].].
> 2022-08-29 17:01:26,096 WARN [c.c.a.AlertManagerImpl]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) No recipients set in global
> setting 'alert.email.addresses', skipping sending alert with subject
> [Console proxy creation failure. zone: xxxxxxxxxx, error details: Unable to
> allocate proxy [Console VM instance {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
> due to [Insufficient address capacity].] and content [Console proxy
> creation failure (zone xxxxxxxxxx)].
> 2022-08-29 17:01:26,096 WARN [c.c.v.SystemVmLoadScanner]
> (consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unexpected exception Unable
> to allocate proxy [Console VM instance {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
> due to [Insufficient address capacity].
> com.cloud.utils.exception.CloudRuntimeException: Unable to allocate proxy
> [Console VM instance {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
> due to [Insufficient address capacity].
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:704)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
> at
> com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
> at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
> at
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
> at
> com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: com.cloud.exception.InsufficientAddressCapacityException:
> Insufficient address capacityScope=interface com.cloud.dc.DataCenter; id=1
> at
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
> at
> com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> ... 38 more
> 2022-08-29 17:01:29,700 DEBUG [o.a.c.h.HAManagerImpl]
> (BackgroundTaskPollManager-5:ctx-994465bd) (logid:1e2822fd) HA health check
> task is running...
> 2022-08-29 17:01:30,444 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 2 routers to
> update status.
> 2022-08-29 17:01:30,449 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 1 VPC's to
> update Redundant State.
> 2022-08-29 17:01:30,451 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 0 networks to
> update RvR status.
> 2022-08-29 17:01:30,459 DEBUG [c.c.a.t.Request]
> (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
> 6-4833769775052102463: Sending { Cmd , MgmtId: 90520740759984, via:
> 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.name":"r-79-VM","router.ip":"169.254.144.198"},"wait":"30","bypassHostMaintenance":"false"}}]
> }
> 2022-08-29 17:01:30,643 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-12:null) (logid:) Seq 6-4833769775052102463:
> Processing: { Ans: , MgmtId: 90520740759984, via: 6, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"PRIMARY","result":"true","details":"Status:
> PRIMARY
> ","wait":"0","bypassHostMaintenance":"false"}}] }
> 2022-08-29 17:01:30,643 DEBUG [c.c.a.t.Request]
> (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
> 6-4833769775052102463: Received: { Ans: , MgmtId: 90520740759984, via:
> 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer } }
> 2022-08-29 17:01:30,643 DEBUG [c.c.a.m.AgentManagerImpl]
> (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Details from
> executing class com.cloud.agent.api.CheckRouterCommand: Status: PRIMARY
>
> 2022-08-29 17:01:30,655 DEBUG [c.c.a.t.Request]
> (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
> 1-5410793477308947401: Sending { Cmd , MgmtId: 90520740759984, via:
> 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.name":"r-80-VM","router.ip":"169.254.217.200"},"wait":"30","bypassHostMaintenance":"false"}}]
> }
> 2022-08-29 17:01:30,796 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-7:null) (logid:) Seq 1-5410793477308947401:
> Processing: { Ans: , MgmtId: 90520740759984, via: 1, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"BACKUP","result":"true","details":"Status:
> BACKUP
> ","wait":"0","bypassHostMaintenance":"false"}}] }
> 2022-08-29 17:01:30,796 DEBUG [c.c.a.t.Request]
> (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
> 1-5410793477308947401: Received: { Ans: , MgmtId: 90520740759984, via:
> 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer } }
> 2022-08-29 17:01:30,796 DEBUG [c.c.a.m.AgentManagerImpl]
> (RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Details from
> executing class com.cloud.agent.api.CheckRouterCommand: Status: BACKUP
>
> 2022-08-29 17:01:33,703 DEBUG [o.a.c.h.HAManagerImpl]
> (BackgroundTaskPollManager-5:ctx-70bca202) (logid:b332dbe0) HA health check
> task is running...
> 2022-08-29 17:01:34,083 DEBUG [c.c.s.StatsCollector]
> (StatsCollector-6:ctx-213e2ecc) (logid:aa0584c9) AutoScaling Monitor is
> running...
> 2022-08-29 17:01:37,707 DEBUG [o.a.c.h.HAManagerImpl]
> (BackgroundTaskPollManager-6:ctx-57493517) (logid:5093e75f) HA health check
> task is running...
> 2022-08-29 17:01:38,640 DEBUG [c.c.s.StatsCollector]
> (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) HostStatsCollector is
> running...
> 2022-08-29 17:01:38,655 DEBUG [c.c.a.t.Request]
> (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq 1-5410793477308947402:
> Received: { Ans: , MgmtId: 90520740759984, via:
> 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> 2022-08-29 17:01:38,707 DEBUG [c.c.a.t.Request]
> (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq 5-8959067033723473395:
> Received: { Ans: , MgmtId: 90520740759984, via:
> 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> 2022-08-29 17:01:38,717 DEBUG [c.c.a.t.Request]
> (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq 6-4833769775052102464:
> Received: { Ans: , MgmtId: 90520740759984, via:
> 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> 2022-08-29 17:01:38,725 DEBUG [c.c.a.t.Request]
> (StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq 7-295548725546192472:
> Received: { Ans: , MgmtId: 90520740759984, via:
> 7(srv-5-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
> 2022-08-29 17:01:41,709 DEBUG [o.a.c.h.HAManagerImpl]
> (BackgroundTaskPollManager-5:ctx-a65ae462) (logid:bc2b7320) HA health check
> task is running...
> 2022-08-29 17:01:45,712 DEBUG [o.a.c.h.HAManagerImpl]
> (BackgroundTaskPollManager-2:ctx-05491ace) (logid:b2232c4d) HA health check
> task is running...
> 2022-08-29 17:01:47,019 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-14:null) (logid:) Ping from Routing host
> 7(srv-5-comp.aaaaaaaa.bbb)
>
> 2022-08-29 17:01:49,715 DEBUG [o.a.c.h.HAManagerImpl]
> (BackgroundTaskPollManager-2:ctx-7943581c) (logid:504c98ff) HA health check
> task is running...
> 2022-08-29 17:01:51,881 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-13:null) (logid:) Ping from Routing host
> 1(srv-2-comp.aaaaaaaa.bbb)
>
> 2022-08-29 17:01:53,718 DEBUG [o.a.c.h.HAManagerImpl]
> (BackgroundTaskPollManager-4:ctx-b11cfa22) (logid:3163613f) HA health check
> task is running...
> 2022-08-29 17:01:53,955 DEBUG [c.c.a.t.Request]
> (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq 1-5410793477308947403:
> Received: { Ans: , MgmtId: 90520740759984, via:
> 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
> 2022-08-29 17:01:54,025 DEBUG [c.c.a.t.Request]
> (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq 5-8959067033723473396:
> Received: { Ans: , MgmtId: 90520740759984, via:
> 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
> 2022-08-29 17:01:54,096 DEBUG [c.c.a.t.Request]
> (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq 6-4833769775052102465:
> Received: { Ans: , MgmtId: 90520740759984, via:
> 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
> 2022-08-29 17:01:54,217 DEBUG [c.c.a.t.Request]
> (StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq 7-295548725546192473:
> Received: { Ans: , MgmtId: 90520740759984, via:
> 7(srv-5-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
> 2022-08-29 17:01:55,765 DEBUG [c.c.s.StatsCollector]
> (secstorage-1:ctx-7d9995d7) (logid:e388a8b8) Verifying image storage [1].
> Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
> 2022-08-29 17:01:55,771 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-7d9995d7) (logid:e388a8b8) Zone [1] is ready to launch
> secondary storage VM.
> 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-413f712d) (logid:440fd101) Zone 1 is ready to launch
> console proxy
> 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-413f712d) (logid:440fd101) Expand console proxy standby
> capacity for zone xxxxxxxxxx
> 2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-413f712d) (logid:440fd101) Allocating console proxy
> standby capacity for zone [1].
> 2022-08-29 17:01:56,034 INFO [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-413f712d) (logid:440fd101) Found a stopped console
> proxy, starting it. Vm id : 85
> 2022-08-29 17:01:56,035 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (consoleproxy-1:ctx-413f712d) (logid:440fd101) start parameter value of
> enterHardwareSetup == <very null> during processing of queued job
> 2022-08-29 17:01:56,044 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (consoleproxy-1:ctx-413f712d) (logid:440fd101) Sync job-4288 execution on
> object VmWorkJobQueue.85
> 2022-08-29 17:01:56,173 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-e75826b1) (logid:e80fe659) Execute sync-queue
> item: SyncQueueItemVO {id:2105, queueId: 2105, contentType: AsyncJob,
> contentId: 4288, lastProcessMsid: 90520740759984, lastprocessNumber: 1,
> lastProcessTime: Mon Aug 29 17:01:56 CEST 2022, created: Mon Aug 29
> 17:01:56 CEST 2022}
> 2022-08-29 17:01:56,174 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-e75826b1) (logid:e80fe659) Schedule queued
> job-4288
> 2022-08-29 17:01:56,179 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:7e671dd0) Add
> job-4288 into job monitoring
> 2022-08-29 17:01:56,186 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5)
> Executing AsyncJobVO: {id:4288, userId: 1, accountId: 1, instanceType:
> null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAVXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 90520740759984, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: Mon Aug 29 17:01:56 CEST 2022, removed:
> null}
> 2022-08-29 17:01:56,186 DEBUG [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5) Run
> VM work job: com.cloud.vm.VmWorkStart for VM 85, job origin: 3445
> 2022-08-29 17:01:56,187 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> (logid:ba10fec5) Execute VM work job:
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> 2022-08-29 17:01:56,189 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> (logid:ba10fec5) orchestrating VM start for 'v-85-VM'
> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
> 2022-08-29 17:01:56,196 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
> transited from [Stopped] to [Starting] with event [StartRequested]. VM's
> original host: null, new host: null, host before state transition: null
> 2022-08-29 17:01:56,196 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> (logid:ba10fec5) Successfully transitioned to start state for VM instance
> {id: "85", name: "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
> type="ConsoleProxy"} reservation id = 0ab68202-d8be-469c-9b2a-87a55c2dc489
> 2022-08-29 17:01:56,200 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts: null
> 2022-08-29 17:01:56,202 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> (logid:ba10fec5) VM start attempt #1
> 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
> 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> (logid:ba10fec5) Trying to allocate a host and storage pools from dc:1,
> pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
> 1073741824
> 2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
> 2022-08-29 17:01:56,212 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
> 2022-08-29 17:01:56,214 DEBUG [c.c.d.FirstFitPlanner]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> (logid:ba10fec5) Searching all possible resources under this Zone: 1
> 2022-08-29 17:01:56,215 DEBUG [c.c.d.FirstFitPlanner]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> (logid:ba10fec5) Listing clusters in order of aggregate capacity, that have
> (at least one host with) enough CPU and RAM capacity under this Zone: 1
> 2022-08-29 17:01:56,218 DEBUG [c.c.d.FirstFitPlanner]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> (logid:ba10fec5) Removing from the clusterId list these clusters from avoid
> set: []
> 2022-08-29 17:01:56,224 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
> 2022-08-29 17:01:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested with
> Custom[UEFI] Boot Type false
> 2022-08-29 17:01:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in dc: 1
> pod:1 cluster:1
> 2022-08-29 17:01:56,226 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4 hosts to
> check for allocation: [Host {"id": "5", "name": "srv-4-comp.aaaaaaaa.bbb",
> "uuid": "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
> {"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb", "uuid":
> "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host {"id": "6",
> "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id": "1",
> "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> 2022-08-29 17:01:56,232 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for allocation
> after prioritization: [Host {"id": "5", "name": "srv-4-comp.aaaaaaaa.bbb",
> "uuid": "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
> {"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb", "uuid":
> "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host {"id": "6",
> "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id": "1",
> "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> 2022-08-29 17:01:56,232 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for speed=500Mhz,
> Ram=1024 MB
> 2022-08-29 17:01:56,232 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
> srv-4-comp.aaaaaaaa.bbb, uuid: 134c625d-15ab-461a-a38d-36fb0b979b5d} is KVM
> hypervisor type, no max guest limit check needed
> 2022-08-29 17:01:56,234 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu capability
> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> 2022-08-29 17:01:56,234 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has enough
> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> cpuOverprovisioningFactor: 1.0
> 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
> and CPU after applying overprovisioning: 127200
> 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 , Requested
> CPU: 500
> 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
> 251009040384 , Requested RAM: (1.00 GB) 1073741824
> 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> available
> 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
> 5, used: 11100, reserved: 0, actual total: 127200, total with
> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> ,considerReservedCapacity?: true
> 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
> 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total: (250.27 GB)
> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> false , considerReservedCapacity?: true
> 2022-08-29 17:01:56,235 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
> list: 5
> 2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
> srv-5-comp.aaaaaaaa.bbb, uuid: 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is KVM
> hypervisor type, no max guest limit check needed
> 2022-08-29 17:01:56,238 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu capability
> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> 2022-08-29 17:01:56,238 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has enough
> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> cpuOverprovisioningFactor: 1.0
> 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
> and CPU after applying overprovisioning: 127200
> 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 , Requested CPU:
> 500
> 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
> 189373743104 , Requested RAM: (1.00 GB) 1073741824
> 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> available
> 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
> 7, used: 90100, reserved: 0, actual total: 127200, total with
> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> ,considerReservedCapacity?: true
> 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
> 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total: (250.27 GB)
> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> false , considerReservedCapacity?: true
> 2022-08-29 17:01:56,240 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
> list: 7
> 2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
> srv-3-comp.aaaaaaaa.bbb, uuid: 6858713b-7e17-46d8-854d-24a13f9c7001} is KVM
> hypervisor type, no max guest limit check needed
> 2022-08-29 17:01:56,242 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu capability
> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> 2022-08-29 17:01:56,242 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has enough
> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> cpuOverprovisioningFactor: 1.0
> 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
> and CPU after applying overprovisioning: 127200
> 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 , Requested CPU:
> 500
> 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
> 204032835584 , Requested RAM: (1.00 GB) 1073741824
> 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> available
> 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
> 6, used: 42900, reserved: 0, actual total: 127200, total with
> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> ,considerReservedCapacity?: true
> 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
> 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total: (250.27 GB)
> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> false , considerReservedCapacity?: true
> 2022-08-29 17:01:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
> list: 6
> 2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
> srv-2-comp.aaaaaaaa.bbb, uuid: 883f7f62-55f2-42f4-8242-16b6847c4204} is KVM
> hypervisor type, no max guest limit check needed
> 2022-08-29 17:01:56,246 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu capability
> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> 2022-08-29 17:01:56,246 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has enough
> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> cpuOverprovisioningFactor: 1.0
> 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
> and CPU after applying overprovisioning: 127200
> 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 , Requested CPU:
> 500
> 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
> 229802639360 , Requested RAM: (1.00 GB) 1073741824
> 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> available
> 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
> 1, used: 48200, reserved: 0, actual total: 127200, total with
> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> ,considerReservedCapacity?: true
> 2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
> 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total: (250.27 GB)
> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> false , considerReservedCapacity?: true
> 2022-08-29 17:01:56,247 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
> list: 1
> 2022-08-29 17:01:56,247 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
> FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator returning 4
> suitable hosts
> 2022-08-29 17:01:56,258 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
> transited from [Starting] to [Stopped] with event [OperationFailed]. VM's
> original host: null, new host: null, host before state transition: null
> 2022-08-29 17:01:56,258 WARN [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> (logid:ba10fec5) Unable to orchestrate start VM instance {id: "85", name:
> "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
> type="ConsoleProxy"} due to [Unable to create deployment, no usable volumes
> found for the VM: 85].
> com.cloud.utils.exception.CloudRuntimeException: Unable to create
> deployment, no usable volumes found for the VM: 85
> at
> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
> at
> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
> at
> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
> at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> 2022-08-29 17:01:56,259 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> (logid:ba10fec5) Done executing VM work job:
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> 2022-08-29 17:01:56,260 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> (logid:ba10fec5) Complete async job-4288, jobStatus: FAILED, resultCode: 0,
> result:
> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODUiLCBuYW1lOiAidi04NS1WTSIsIHV1aWQ6ICIxMTBkODYyMC1jMDI5LTRjYTktOTA4NC0yMTAwOTk3MjVhODAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODVdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
> 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> (logid:ba10fec5) Publish async job-4288 complete on message bus
> 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> (logid:ba10fec5) Wake up jobs related to job-4288
> 2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> (logid:ba10fec5) Update db status for job-4288
> 2022-08-29 17:01:56,262 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
> (logid:ba10fec5) Wake up jobs joined with job-4288 and disjoin all subjobs
> created from job- 4288
> 2022-08-29 17:01:56,270 DEBUG [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5)
> Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 85, job
> origin: 3445
> 2022-08-29 17:01:56,270 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5)
> Done executing com.cloud.vm.VmWorkStart for job-4288
> 2022-08-29 17:01:56,272 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5)
> Remove job-4288 from job monitoring
> 2022-08-29 17:01:56,277 WARN [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-413f712d) (logid:440fd101) Unable to allocate console
> proxy standby capacity for zone [1] due to [Unable to orchestrate start VM
> instance {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to [Unable
> to create deployment, no usable volumes found for the VM: 85].].
> com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate
> start VM instance {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to [Unable
> to create deployment, no usable volumes found for the VM: 85].
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> 2022-08-29 17:01:56,277 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> (consoleproxy-1:ctx-413f712d) (logid:440fd101) received console proxy alert
> 2022-08-29 17:01:56,278 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> (consoleproxy-1:ctx-413f712d) (logid:440fd101) Console proxy creation
> failure, zone: xxxxxxxxxx
> 2022-08-29 17:01:56,279 WARN [c.c.a.AlertManagerImpl]
> (consoleproxy-1:ctx-413f712d) (logid:440fd101) alertType=[10]
> dataCenterId=[1] podId=[null] clusterId=[null] message=[Console proxy
> creation failure. zone: xxxxxxxxxx, error details: Unable to orchestrate
> start VM instance {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to [Unable
> to create deployment, no usable volumes found for the VM: 85].].
> 2022-08-29 17:01:56,284 WARN [c.c.a.AlertManagerImpl]
> (consoleproxy-1:ctx-413f712d) (logid:440fd101) No recipients set in global
> setting 'alert.email.addresses', skipping sending alert with subject
> [Console proxy creation failure. zone: xxxxxxxxxx, error details: Unable to
> orchestrate start VM instance {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to [Unable
> to create deployment, no usable volumes found for the VM: 85].] and content
> [Console proxy creation failure (zone xxxxxxxxxx)].
> 2022-08-29 17:01:56,284 WARN [c.c.v.SystemVmLoadScanner]
> (consoleproxy-1:ctx-413f712d) (logid:440fd101) Unexpected exception Unable
> to orchestrate start VM instance {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to [Unable
> to create deployment, no usable volumes found for the VM: 85].
> com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate
> start VM instance {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to [Unable
> to create deployment, no usable volumes found for the VM: 85].
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> 2022-08-29 17:01:57,721 DEBUG [o.a.c.h.HAManagerImpl]
> (BackgroundTaskPollManager-4:ctx-9528226f) (logid:4512ff9b) HA health check
> task is running...
> 2022-08-29 17:01:59,504 DEBUG [c.c.s.StatsCollector]
> (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) StorageCollector is
> running...
> 2022-08-29 17:01:59,518 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) getCommandHostDelegation:
> class com.cloud.agent.api.GetStorageStatsCommand
> 2022-08-29 17:01:59,518 DEBUG [c.c.h.XenServerGuru]
> (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning the
> default host to execute commands because the command is not of Copy type.
> 2022-08-29 17:01:59,571 DEBUG [c.c.a.t.Request]
> (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
> 11-7523826127475836583: Received: { Ans: , MgmtId: 90520740759984, via:
> 11(s-32-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> 2022-08-29 17:01:59,574 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) getCommandHostDelegation:
> class com.cloud.agent.api.GetStorageStatsCommand
> 2022-08-29 17:01:59,574 DEBUG [c.c.h.XenServerGuru]
> (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning the
> default host to execute commands because the command is not of Copy type.
> 2022-08-29 17:01:59,615 DEBUG [c.c.a.t.Request]
> (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq 5-8959067033723473397:
> Received: { Ans: , MgmtId: 90520740759984, via:
> 5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> 2022-08-29 17:01:59,624 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) getCommandHostDelegation:
> class com.cloud.agent.api.GetStorageStatsCommand
> 2022-08-29 17:01:59,624 DEBUG [c.c.h.XenServerGuru]
> (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning the
> default host to execute commands because the command is not of Copy type.
> 2022-08-29 17:01:59,653 DEBUG [c.c.a.t.Request]
> (StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq 6-4833769775052102466:
> Received: { Ans: , MgmtId: 90520740759984, via:
> 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> 2022-08-29 17:01:59,979 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-12:null) (logid:) Ping from Routing host
> 5(srv-4-comp.aaaaaaaa.bbb)
>
> 2022-08-29 17:02:00,269 DEBUG [c.c.s.StatsCollector]
> (StatsCollector-5:ctx-d2b6ffd3) (logid:95201678) DbCollector is running...
> 2022-08-29 17:02:00,443 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 2 routers to
> update status.
> 2022-08-29 17:02:00,449 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 1 VPC's to
> update Redundant State.
> 2022-08-29 17:02:00,450 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 0 networks to
> update RvR status.
> 2022-08-29 17:02:00,457 DEBUG [c.c.a.t.Request]
> (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
> 6-4833769775052102467: Sending { Cmd , MgmtId: 90520740759984, via:
> 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.name":"r-79-VM","router.ip":"169.254.144.198"},"wait":"30","bypassHostMaintenance":"false"}}]
> }
> 2022-08-29 17:02:00,599 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-7:null) (logid:) Seq 6-4833769775052102467:
> Processing: { Ans: , MgmtId: 90520740759984, via: 6, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"PRIMARY","result":"true","details":"Status:
> PRIMARY
> ","wait":"0","bypassHostMaintenance":"false"}}] }
> 2022-08-29 17:02:00,599 DEBUG [c.c.a.t.Request]
> (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
> 6-4833769775052102467: Received: { Ans: , MgmtId: 90520740759984, via:
> 6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer } }
> 2022-08-29 17:02:00,599 DEBUG [c.c.a.m.AgentManagerImpl]
> (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Details
> from executing class com.cloud.agent.api.CheckRouterCommand: Status: PRIMARY
>
> 2022-08-29 17:02:00,612 DEBUG [c.c.a.t.Request]
> (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
> 1-5410793477308947404: Sending { Cmd , MgmtId: 90520740759984, via:
> 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.name":"r-80-VM","router.ip":"169.254.217.200"},"wait":"30","bypassHostMaintenance":"false"}}]
> }
> 2022-08-29 17:02:00,748 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-4:null) (logid:) Ping from 11(s-32-VM)
> 2022-08-29 17:02:00,750 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-1:null) (logid:) Seq 1-5410793477308947404:
> Processing: { Ans: , MgmtId: 90520740759984, via: 1, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.CheckRouterAnswer":{"state":"BACKUP","result":"true","details":"Status:
> BACKUP
> ","wait":"0","bypassHostMaintenance":"false"}}] }
> 2022-08-29 17:02:00,750 DEBUG [c.c.a.t.Request]
> (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
> 1-5410793477308947404: Received: { Ans: , MgmtId: 90520740759984, via:
> 1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer } }
> 2022-08-29 17:02:00,751 DEBUG [c.c.a.m.AgentManagerImpl]
> (RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Details
> from executing class com.cloud.agent.api.CheckRouterCommand: Status: BACKUP
>
> 2022-08-29 17:02:01,362 DEBUG [c.c.s.StatsCollector]
> (Cluster-Worker-839:ctx-e097a8ab) (logid:3deacf65) StatusUpdate from
> 90520740759984, json:
> {"managementServerHostId":1,"managementServerHostUuid":"5615ca5f-0716-4da1-93ec-31f79fd19bbe","collectionTime":"Aug
> 29, 2022, 5:02:01
> PM","sessions":1,"cpuUtilization":0.0,"totalJvmMemoryBytes":324534272,"freeJvmMemoryBytes":68595424,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":103760388,"jvmStartTime":1661681560910,"availableProcessors":6,"loadAverage":0.02,"totalInit":139788288,"totalUsed":474791104,"totalCommitted":557662208,"pid":1522332,"jvmName":"1522332@vm-1-cs-mgmt.aaaaaaaa.bbb","jvmVendor":"Ubuntu","jvmVersion":"11.0.16+8-post-Ubuntu-0ubuntu120.04","osDistribution":"Ubuntu
> 20.04.5
> LTS","agentCount":5,"heapMemoryUsed":256775312,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":13,"threadsRunnableCount":15,"threadsTerminatedCount":0,"threadsTotalCount":420,"threadsWaitingCount":331,"systemMemoryTotal":8343236608,"systemMemoryFree":2623193088,"systemMemoryUsed":814808,"systemMemoryVirtualSize":7494184960,"logInfo":"/var/log/cloudstack/management/management-server.log
> using: 74M\non disk /dev/vda2 mounted on / (14%
> full)","systemTotalCpuCycles":15870.180000000002,"systemLoadAverages":[0.02,0.03,0.0],"systemCyclesUsage":[4116569,2615101,2716460424],"dbLocal":false,"usageLocal":false,"systemBootTime":"Jul
> 8, 2022, 2:35:55 AM","kernelVersion":"5.4.0-121-generic"}
> 2022-08-29 17:02:01,723 DEBUG [o.a.c.h.HAManagerImpl]
> (BackgroundTaskPollManager-4:ctx-7abb1b5d) (logid:2320950d) HA health check
> task is running...
> 2022-08-29 17:02:05,417 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Vm-Operations-Cleanup-1:ctx-e741e208) (logid:3f5129e0) VM Operation Thread
> Running
> 2022-08-29 17:02:05,725 DEBUG [o.a.c.h.HAManagerImpl]
> (BackgroundTaskPollManager-4:ctx-8e7f6766) (logid:710cb3b2) HA health check
> task is running...
> 2022-08-29 17:02:08,626 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-11:null) (logid:) Ping from Routing host
> 6(srv-3-comp.aaaaaaaa.bbb)
>
> 2022-08-29 17:02:09,727 DEBUG [o.a.c.h.HAManagerImpl]
> (BackgroundTaskPollManager-4:ctx-26a5ff07) (logid:09b3c89a) HA health check
> task is running...
> 2022-08-29 17:02:13,729 DEBUG [o.a.c.h.HAManagerImpl]
> (BackgroundTaskPollManager-4:ctx-b4d4e374) (logid:9048c7a7) HA health check
> task is running...
> 2022-08-29 17:02:17,731 DEBUG [o.a.c.h.HAManagerImpl]
> (BackgroundTaskPollManager-4:ctx-3db3d208) (logid:7bb73f7b) HA health check
> task is running...
> 2022-08-29 17:02:21,734 DEBUG [o.a.c.h.HAManagerImpl]
> (BackgroundTaskPollManager-1:ctx-fca1d248) (logid:1ed5381a) HA health check
> task is running...
> 2022-08-29 17:02:25,737 DEBUG [o.a.c.h.HAManagerImpl]
> (BackgroundTaskPollManager-1:ctx-984e2382) (logid:e6c3d724) HA health check
> task is running...
> 2022-08-29 17:02:25,759 DEBUG [c.c.s.StatsCollector]
> (secstorage-1:ctx-64c610ed) (logid:6603cd38) Verifying image storage [1].
> Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
> 2022-08-29 17:02:25,763 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-64c610ed) (logid:6603cd38) Zone [1] is ready to launch
> secondary storage VM.
> 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Zone 1 is ready to launch
> console proxy
> 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Expand console proxy standby
> capacity for zone xxxxxxxxxx
> 2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Allocating console proxy
> standby capacity for zone [1].
> 2022-08-29 17:02:26,030 INFO [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Found a stopped console
> proxy, starting it. Vm id : 85
> 2022-08-29 17:02:26,031 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) start parameter value of
> enterHardwareSetup == <very null> during processing of queued job
> 2022-08-29 17:02:26,037 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Sync job-4289 execution on
> object VmWorkJobQueue.85
> 2022-08-29 17:02:26,167 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-bba6b89d) (logid:f9aed0eb) Execute sync-queue
> item: SyncQueueItemVO {id:2106, queueId: 2105, contentType: AsyncJob,
> contentId: 4289, lastProcessMsid: 90520740759984, lastprocessNumber: 2,
> lastProcessTime: Mon Aug 29 17:02:26 CEST 2022, created: Mon Aug 29
> 17:02:26 CEST 2022}
> 2022-08-29 17:02:26,167 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-bba6b89d) (logid:f9aed0eb) Schedule queued
> job-4289
> 2022-08-29 17:02:26,172 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:f919f9b3) Add
> job-4289 into job monitoring
> 2022-08-29 17:02:26,181 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5)
> Executing AsyncJobVO: {id:4289, userId: 1, accountId: 1, instanceType:
> null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAVXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 90520740759984, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: Mon Aug 29 17:02:26 CEST 2022, removed:
> null}
> 2022-08-29 17:02:26,181 DEBUG [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5) Run
> VM work job: com.cloud.vm.VmWorkStart for VM 85, job origin: 3445
> 2022-08-29 17:02:26,183 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> (logid:ba10fec5) Execute VM work job:
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> 2022-08-29 17:02:26,184 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> (logid:ba10fec5) orchestrating VM start for 'v-85-VM'
> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
> 2022-08-29 17:02:26,192 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
> transited from [Stopped] to [Starting] with event [StartRequested]. VM's
> original host: null, new host: null, host before state transition: null
> 2022-08-29 17:02:26,192 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> (logid:ba10fec5) Successfully transitioned to start state for VM instance
> {id: "85", name: "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
> type="ConsoleProxy"} reservation id = 24bf955f-96d8-4f4f-ae33-2cae2f3e0425
> 2022-08-29 17:02:26,198 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts: null
> 2022-08-29 17:02:26,200 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> (logid:ba10fec5) VM start attempt #1
> 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
> 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> (logid:ba10fec5) Trying to allocate a host and storage pools from dc:1,
> pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
> 1073741824
> 2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
> 2022-08-29 17:02:26,211 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
> 2022-08-29 17:02:26,212 DEBUG [c.c.d.FirstFitPlanner]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> (logid:ba10fec5) Searching all possible resources under this Zone: 1
> 2022-08-29 17:02:26,213 DEBUG [c.c.d.FirstFitPlanner]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> (logid:ba10fec5) Listing clusters in order of aggregate capacity, that have
> (at least one host with) enough CPU and RAM capacity under this Zone: 1
> 2022-08-29 17:02:26,216 DEBUG [c.c.d.FirstFitPlanner]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> (logid:ba10fec5) Removing from the clusterId list these clusters from avoid
> set: []
> 2022-08-29 17:02:26,221 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
> 2022-08-29 17:02:26,222 INFO [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested with
> Custom[UEFI] Boot Type false
> 2022-08-29 17:02:26,223 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in dc: 1
> pod:1 cluster:1
> 2022-08-29 17:02:26,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4 hosts to
> check for allocation: [Host {"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb",
> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
> {"id": "1", "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}, Host {"id": "6",
> "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id": "5",
> "name": "srv-4-comp.aaaaaaaa.bbb", "uuid":
> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}]
> 2022-08-29 17:02:26,231 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for allocation
> after prioritization: [Host {"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb",
> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
> {"id": "1", "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}, Host {"id": "6",
> "name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id": "5",
> "name": "srv-4-comp.aaaaaaaa.bbb", "uuid":
> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}]
> 2022-08-29 17:02:26,231 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for speed=500Mhz,
> Ram=1024 MB
> 2022-08-29 17:02:26,231 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
> srv-5-comp.aaaaaaaa.bbb, uuid: 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is KVM
> hypervisor type, no max guest limit check needed
> 2022-08-29 17:02:26,234 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu capability
> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> 2022-08-29 17:02:26,234 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has enough
> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> cpuOverprovisioningFactor: 1.0
> 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
> and CPU after applying overprovisioning: 127200
> 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 , Requested CPU:
> 500
> 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
> 189373743104 , Requested RAM: (1.00 GB) 1073741824
> 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> available
> 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
> 7, used: 90100, reserved: 0, actual total: 127200, total with
> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> ,considerReservedCapacity?: true
> 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
> 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total: (250.27 GB)
> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> false , considerReservedCapacity?: true
> 2022-08-29 17:02:26,236 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
> list: 7
> 2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
> srv-2-comp.aaaaaaaa.bbb, uuid: 883f7f62-55f2-42f4-8242-16b6847c4204} is KVM
> hypervisor type, no max guest limit check needed
> 2022-08-29 17:02:26,239 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu capability
> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> 2022-08-29 17:02:26,239 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has enough
> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> cpuOverprovisioningFactor: 1.0
> 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
> and CPU after applying overprovisioning: 127200
> 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 , Requested CPU:
> 500
> 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
> 229802639360 , Requested RAM: (1.00 GB) 1073741824
> 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> available
> 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
> 1, used: 48200, reserved: 0, actual total: 127200, total with
> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> ,considerReservedCapacity?: true
> 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
> 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total: (250.27 GB)
> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> false , considerReservedCapacity?: true
> 2022-08-29 17:02:26,240 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
> list: 1
> 2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
> srv-3-comp.aaaaaaaa.bbb, uuid: 6858713b-7e17-46d8-854d-24a13f9c7001} is KVM
> hypervisor type, no max guest limit check needed
> 2022-08-29 17:02:26,243 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu capability
> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> 2022-08-29 17:02:26,243 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has enough
> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> cpuOverprovisioningFactor: 1.0
> 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
> and CPU after applying overprovisioning: 127200
> 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 , Requested CPU:
> 500
> 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
> 204032835584 , Requested RAM: (1.00 GB) 1073741824
> 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> available
> 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
> 6, used: 42900, reserved: 0, actual total: 127200, total with
> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> ,considerReservedCapacity?: true
> 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
> 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total: (250.27 GB)
> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> false , considerReservedCapacity?: true
> 2022-08-29 17:02:26,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
> list: 6
> 2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
> srv-4-comp.aaaaaaaa.bbb, uuid: 134c625d-15ab-461a-a38d-36fb0b979b5d} is KVM
> hypervisor type, no max guest limit check needed
> 2022-08-29 17:02:26,247 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu capability
> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> 2022-08-29 17:02:26,247 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has enough
> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> cpuOverprovisioningFactor: 1.0
> 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
> and CPU after applying overprovisioning: 127200
> 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 , Requested
> CPU: 500
> 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
> 251009040384 , Requested RAM: (1.00 GB) 1073741824
> 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> available
> 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
> 5, used: 11100, reserved: 0, actual total: 127200, total with
> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> ,considerReservedCapacity?: true
> 2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
> 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total: (250.27 GB)
> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> false , considerReservedCapacity?: true
> 2022-08-29 17:02:26,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
> list: 5
> 2022-08-29 17:02:26,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
> FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator returning 4
> suitable hosts
> 2022-08-29 17:02:26,260 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> (logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
> transited from [Starting] to [Stopped] with event [OperationFailed]. VM's
> original host: null, new host: null, host before state transition: null
> 2022-08-29 17:02:26,260 WARN [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> (logid:ba10fec5) Unable to orchestrate start VM instance {id: "85", name:
> "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
> type="ConsoleProxy"} due to [Unable to create deployment, no usable volumes
> found for the VM: 85].
> com.cloud.utils.exception.CloudRuntimeException: Unable to create
> deployment, no usable volumes found for the VM: 85
> at
> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
> at
> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
> at
> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
> at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> 2022-08-29 17:02:26,261 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> (logid:ba10fec5) Done executing VM work job:
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
> 2022-08-29 17:02:26,262 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> (logid:ba10fec5) Complete async job-4289, jobStatus: FAILED, resultCode: 0,
> result:
> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODUiLCBuYW1lOiAidi04NS1WTSIsIHV1aWQ6ICIxMTBkODYyMC1jMDI5LTRjYTktOTA4NC0yMTAwOTk3MjVhODAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODVdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
> 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> (logid:ba10fec5) Publish async job-4289 complete on message bus
> 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> (logid:ba10fec5) Wake up jobs related to job-4289
> 2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> (logid:ba10fec5) Update db status for job-4289
> 2022-08-29 17:02:26,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
> (logid:ba10fec5) Wake up jobs joined with job-4289 and disjoin all subjobs
> created from job- 4289
> 2022-08-29 17:02:26,269 DEBUG [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5)
> Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 85, job
> origin: 3445
> 2022-08-29 17:02:26,270 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5)
> Done executing com.cloud.vm.VmWorkStart for job-4289
> 2022-08-29 17:02:26,271 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5)
> Remove job-4289 from job monitoring
> 2022-08-29 17:02:26,277 WARN [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Unable to allocate console
> proxy standby capacity for zone [1] due to [Unable to orchestrate start VM
> instance {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to [Unable
> to create deployment, no usable volumes found for the VM: 85].].
> com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate
> start VM instance {id: "85", name: "v-85-VM", uuid:
> "110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to [Unable
> to create deployment, no usable volumes found for the VM: 85].
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> 2022-08-29 17:02:26,277 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> (consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) received console proxy alert
>
>
>
> Am Mo., 29. Aug. 2022 um 16:30 Uhr schrieb Slavka Peleva <
> slavkap@storpool.com>:
>
>> Thanks, Chris, for the information!
>>
>> Are you using custom offerings for the system VMs? Can you share more
>> info if yes?
>>
>> About the log, it's not the complete log from where the deployment
>> started. The deployment begins a bit earlier than what you shared. Can you
>> search the management log for `Sync job-3539 execution on object
>> VmWorkJobQueue` to get the `logid` for the volume creation (coloured in
>> blue)? And if possible, share the logs with the found log ID
>>
>> Here is an example from my logs:
>>
>> 2022-08-29 17:07:49,114 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (consoleproxy-1:ctx-a952a3dc) (logid:89157bb9) Sync job-60484 execution
>> on object VmWorkJobQueue.4904
>>
>> 2022-08-29 17:07:50,816 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>> (Work-Job-Executor-5:ctx-56172581 job-60479/job-60484 ctx-a900986d)
>> (logid:95d750ae) Execute VM work job:
>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":4904,"handlerName":"VirtualMachineManagerImpl"}
>> 2022-08-29 17:07:50,817 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> (Work-Job-Executor-5:ctx-56172581 job-60479/job-60484 ctx-a900986d)
>> (logid:95d750ae) orchestrating VM start for 'v-4904-VM'
>> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
>>
>>
>> On Mon, Aug 29, 2022 at 2:29 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
>>
>>> Hi Slavka,
>>>
>>> thats what i did after your suggestion / informatin and where the error
>>> occured.
>>> Deleted / Destroyed the consoleproxy and waited till recreation. Which
>>> didn't worked throwing the error i pasted above.
>>> Sorry for beeing so unspecific.
>>>
>>> Regards,
>>> Chris
>>>
>>> Am Mo., 29. Aug. 2022 um 12:46 Uhr schrieb Slavka Peleva <
>>> slavkap@storpool.com>:
>>>
>>>> Hi Chris,
>>>>
>>>> I mean to recreate the system VMs by destroying them.
>>>>
>>>> On Mon, Aug 29, 2022 at 12:38 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
>>>>
>>>>> Hi Slavka,
>>>>>
>>>>> didn't tried to restart the SystemVMs. Reading through the docs i had
>>>>> the impression that "live-patching" would be enough for this....
>>>>> Anyway - I tried my luck with the console proxy.
>>>>>
>>>>> Which failed utterly. Not starting as it isn't finding any useable
>>>>> storage pool. - logs at the end of the mail.
>>>>>
>>>>> I tried deploying a VM using the same storage pool which worked
>>>>> wonderfully.
>>>>> There i noticed that all my custome storage-offerings are unuseable at
>>>>> the moment. They are not accepting any custome storage sizes anymore...
>>>>> Throwing failures "VM Creation failed. Volume size: -1GB is out of allowed
>>>>> tange. Max:3072 Min:1"...
>>>>> So this is also currently broken....
>>>>>
>>>>> 2022-08-29 10:51:56,189 DEBUG [c.c.v.VmWorkJobDispatcher]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5) Run
>>>>> VM work job: com.cloud.vm.VmWorkStart for VM 83, job origin: 3445
>>>>> 2022-08-29 10:51:56,191 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>>> (logid:ba10fec5) Execute VM work job:
>>>>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
>>>>> 2022-08-29 10:51:56,192 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>>> (logid:ba10fec5) orchestrating VM start for 'v-83-VM'
>>>>> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
>>>>> 2022-08-29 10:51:56,198 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
>>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} state
>>>>> transited from [Stopped] to [Starting] with event [StartRequested]. VM's
>>>>> original host: null, new host: null, host before state transition: null
>>>>> 2022-08-29 10:51:56,198 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>>> (logid:ba10fec5) Successfully transitioned to start state for VM instance
>>>>> {id: "83", name: "v-83-VM", uuid: "b1d1f9c6-eafe-4527-9335-e649c646aab0",
>>>>> type="ConsoleProxy"} reservation id = 6641ccc0-bcef-4b1a-a1f7-8dd8000626ad
>>>>> 2022-08-29 10:51:56,203 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>>> (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts: null
>>>>> 2022-08-29 10:51:56,206 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>>> (logid:ba10fec5) VM start attempt #1
>>>>> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>>> (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
>>>>> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>>> (logid:ba10fec5) Trying to allocate a host and storage pools from dc:1,
>>>>> pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
>>>>> 1073741824
>>>>> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>>> (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
>>>>> 2022-08-29 10:51:56,214 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>>> (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
>>>>> 2022-08-29 10:51:56,215 DEBUG [c.c.d.FirstFitPlanner]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>>> (logid:ba10fec5) Searching all possible resources under this Zone: 1
>>>>> 2022-08-29 10:51:56,216 DEBUG [c.c.d.FirstFitPlanner]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>>> (logid:ba10fec5) Listing clusters in order of aggregate capacity, that have
>>>>> (at least one host with) enough CPU and RAM capacity under this Zone: 1
>>>>> 2022-08-29 10:51:56,219 DEBUG [c.c.d.FirstFitPlanner]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>>> (logid:ba10fec5) Removing from the clusterId list these clusters from avoid
>>>>> set: []
>>>>> 2022-08-29 10:51:56,224 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>>> (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
>>>>> 2022-08-29 10:51:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested with
>>>>> Custom[UEFI] Boot Type false
>>>>> 2022-08-29 10:51:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in dc: 1
>>>>> pod:1 cluster:1
>>>>> 2022-08-29 10:51:56,227 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4 hosts to
>>>>> check for allocation: [Host {"id": "7", "name": "srv-5-comp.blabla.blub",
>>>>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
>>>>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
>>>>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host {"id": "6",
>>>>> "name": "srv-3-comp.blabla.blub", "uuid":
>>>>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id": "1",
>>>>> "name": "srv-2-comp.blabla.blub", "uuid":
>>>>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
>>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for allocation
>>>>> after prioritization: [Host {"id": "7", "name": "srv-5-comp.blabla.blub",
>>>>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
>>>>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
>>>>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host {"id": "6",
>>>>> "name": "srv-3-comp.blabla.blub", "uuid":
>>>>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id": "1",
>>>>> "name": "srv-2-comp.blabla.blub", "uuid":
>>>>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
>>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for speed=500Mhz,
>>>>> Ram=1024 MB
>>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
>>>>> srv-5-comp.blabla.blub, uuid: 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is KVM
>>>>> hypervisor type, no max guest limit check needed
>>>>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu capability
>>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
>>>>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has enough
>>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
>>>>> cpuOverprovisioningFactor: 1.0
>>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
>>>>> and CPU after applying overprovisioning: 127200
>>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 , Requested CPU:
>>>>> 500
>>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
>>>>> 189373743104 , Requested RAM: (1.00 GB) 1073741824
>>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
>>>>> available
>>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
>>>>> 7, used: 90100, reserved: 0, actual total: 127200, total with
>>>>> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
>>>>> ,considerReservedCapacity?: true
>>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
>>>>> 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total: (250.27 GB)
>>>>> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
>>>>> false , considerReservedCapacity?: true
>>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
>>>>> list: 7
>>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
>>>>> srv-4-comp.blabla.blub, uuid: 134c625d-15ab-461a-a38d-36fb0b979b5d} is KVM
>>>>> hypervisor type, no max guest limit check needed
>>>>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu capability
>>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
>>>>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has enough
>>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
>>>>> cpuOverprovisioningFactor: 1.0
>>>>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
>>>>> and CPU after applying overprovisioning: 127200
>>>>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 , Requested
>>>>> CPU: 500
>>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
>>>>> 251009040384 , Requested RAM: (1.00 GB) 1073741824
>>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
>>>>> available
>>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
>>>>> 5, used: 11100, reserved: 0, actual total: 127200, total with
>>>>> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
>>>>> ,considerReservedCapacity?: true
>>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
>>>>> 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total: (250.27 GB)
>>>>> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
>>>>> false , considerReservedCapacity?: true
>>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
>>>>> list: 5
>>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
>>>>> srv-3-comp.blabla.blub, uuid: 6858713b-7e17-46d8-854d-24a13f9c7001} is KVM
>>>>> hypervisor type, no max guest limit check needed
>>>>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu capability
>>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
>>>>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has enough
>>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
>>>>> cpuOverprovisioningFactor: 1.0
>>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
>>>>> and CPU after applying overprovisioning: 127200
>>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 , Requested CPU:
>>>>> 500
>>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
>>>>> 204032835584 , Requested RAM: (1.00 GB) 1073741824
>>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
>>>>> available
>>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
>>>>> 6, used: 42900, reserved: 0, actual total: 127200, total with
>>>>> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
>>>>> ,considerReservedCapacity?: true
>>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
>>>>> 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total: (250.27 GB)
>>>>> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
>>>>> false , considerReservedCapacity?: true
>>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
>>>>> list: 6
>>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
>>>>> srv-2-comp.blabla.blub, uuid: 883f7f62-55f2-42f4-8242-16b6847c4204} is KVM
>>>>> hypervisor type, no max guest limit check needed
>>>>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu capability
>>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
>>>>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has enough
>>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
>>>>> cpuOverprovisioningFactor: 1.0
>>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
>>>>> and CPU after applying overprovisioning: 127200
>>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 , Requested CPU:
>>>>> 500
>>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
>>>>> 229802639360 , Requested RAM: (1.00 GB) 1073741824
>>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
>>>>> available
>>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
>>>>> 1, used: 48200, reserved: 0, actual total: 127200, total with
>>>>> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
>>>>> ,considerReservedCapacity?: true
>>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
>>>>> 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total: (250.27 GB)
>>>>> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
>>>>> false , considerReservedCapacity?: true
>>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
>>>>> list: 1
>>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator returning 4
>>>>> suitable hosts
>>>>> 2022-08-29 10:51:56,260 DEBUG [c.c.c.CapacityManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
>>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} state
>>>>> transited from [Starting] to [Stopped] with event [OperationFailed]. VM's
>>>>> original host: null, new host: null, host before state transition: null
>>>>> 2022-08-29 10:51:56,260 WARN [c.c.v.VirtualMachineManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>>> (logid:ba10fec5) Unable to orchestrate start VM instance {id: "83", name:
>>>>> "v-83-VM", uuid: "b1d1f9c6-eafe-4527-9335-e649c646aab0",
>>>>> type="ConsoleProxy"} due to [Unable to create deployment, no usable volumes
>>>>> found for the VM: 83].
>>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to create
>>>>> deployment, no usable volumes found for the VM: 83
>>>>> at
>>>>> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
>>>>> at
>>>>> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
>>>>> at
>>>>> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
>>>>> at
>>>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
>>>>> at
>>>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
>>>>> at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>>>>> Source)
>>>>> at
>>>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>>>> at
>>>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>>>> at
>>>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>>>>> at
>>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>>>> at
>>>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>>>>> at
>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>>>> at
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>>>> at
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>>>> at
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>>>> at
>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>>>> at
>>>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>>>>> at
>>>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>>>> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>>>> at
>>>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>> at
>>>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>> 2022-08-29 10:51:56,262 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>>> (logid:ba10fec5) Done executing VM work job:
>>>>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
>>>>> 2022-08-29 10:51:56,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>>> (logid:ba10fec5) Complete async job-3539, jobStatus: FAILED, resultCode: 0,
>>>>> result:
>>>>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODMiLCBuYW1lOiAidi04My1WTSIsIHV1aWQ6ICJiMWQxZjljNi1lYWZlLTQ1MjctOTMzNS1lNjQ5YzY0NmFhYjAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODNdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
>>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>>> (logid:ba10fec5) Publish async job-3539 complete on message bus
>>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>>> (logid:ba10fec5) Wake up jobs related to job-3539
>>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>>> (logid:ba10fec5) Update db status for job-3539
>>>>> 2022-08-29 10:51:56,266 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>>> (logid:ba10fec5) Wake up jobs joined with job-3539 and disjoin all subjobs
>>>>> created from job- 3539
>>>>> 2022-08-29 10:51:56,276 DEBUG [c.c.v.VmWorkJobDispatcher]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5) Done
>>>>> with run of VM work job: com.cloud.vm.VmWorkStart for VM 83, job origin:
>>>>> 3445
>>>>> 2022-08-29 10:51:56,276 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5) Done
>>>>> executing com.cloud.vm.VmWorkStart for job-3539
>>>>> 2022-08-29 10:51:56,278 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5)
>>>>> Remove job-3539 from job monitoring
>>>>> 2022-08-29 10:51:56,285 WARN [c.c.c.ConsoleProxyManagerImpl]
>>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Unable to allocate console
>>>>> proxy standby capacity for zone [1] due to [Unable to orchestrate start VM
>>>>> instance {id: "83", name: "v-83-VM", uuid:
>>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to [Unable
>>>>> to create deployment, no usable volumes found for the VM: 83].].
>>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate
>>>>> start VM instance {id: "83", name: "v-83-VM", uuid:
>>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to [Unable
>>>>> to create deployment, no usable volumes found for the VM: 83].
>>>>> at
>>>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
>>>>> at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>>>>> Source)
>>>>> at
>>>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>>>> at
>>>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>>>> at
>>>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>>>>> at
>>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>>>> at
>>>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>>>>> at
>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>>>> at
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>>>> at
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>>>> at
>>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>>>> at
>>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>>>> at
>>>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>>>>> at
>>>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>>>> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>>>> at
>>>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>>> at
>>>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>>> 2022-08-29 10:51:56,287 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) received console proxy alert
>>>>> 2022-08-29 10:51:56,288 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Console proxy creation
>>>>> failure, zone: blabla
>>>>> 2022-08-29 10:51:56,289 WARN [c.c.a.AlertManagerImpl]
>>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) alertType=[10]
>>>>> dataCenterId=[1] podId=[null] clusterId=[null] message=[Console proxy
>>>>> creation failure. zone: blabla, error details: Unable to orchestrate start
>>>>> VM instance {id: "83", name: "v-83-VM", uuid:
>>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to [Unable
>>>>> to create deployment, no usable volumes found for the VM: 83].].
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Am Mo., 29. Aug. 2022 um 09:40 Uhr schrieb Slavka Peleva
>>>>> <sl...@storpool.com.invalid>:
>>>>>
>>>>>> Hi Chris,
>>>>>>
>>>>>> Did you recreate the system VMs? In the 4.17 version, the
>>>>>> `systemvm.iso` is
>>>>>> deprecated. In its place, you should have the file on agents
>>>>>> `/usr/share/cloudstack-common/vms/agent.zip`.
>>>>>> Can you share the complete log if the system VMs are newly created
>>>>>> after
>>>>>> the upgrade?
>>>>>>
>>>>>> Best regards,
>>>>>> Slavka
>>>>>>
>>>>>> On Sun, Aug 28, 2022 at 2:16 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
>>>>>>
>>>>>> > Hi everyone,
>>>>>> >
>>>>>> > faceing some challanges again after upgradeing to 4.17.
>>>>>> > Did as explained in the docs.
>>>>>> >
>>>>>> > Afterwards I am now not able to migrate system VMs to different
>>>>>> hosts.
>>>>>> > Getting the error
>>>>>> > "Exception during migrate: org.libvirt.LibvirtException: Cannot
>>>>>> access
>>>>>> > storage file '/usr/share/cloudstack-common/vms/systemvm.iso': No
>>>>>> such file
>>>>>> > or directory"
>>>>>> >
>>>>>> > Checked the exitstence of the file on the servers:
>>>>>> > Mangement-Server has a "systemvm.iso.bak"
>>>>>> > Hosts - no "systemvm.iso" at all present.
>>>>>> >
>>>>>> > Maybe someone has an idea on how to move on with this?
>>>>>> >
>>>>>> > Regards,
>>>>>> > Chris
>>>>>> >
>>>>>>
>>>>>
Re: Error migrate System VMs after upgrageing to 4.17
Posted by "Vash_X@gmx.de" <Va...@gmx.de>.
Hi Slavka,
no, i am not using any kind of custome offerings.
I try to provide a more detailled log this time.
Situation:
v-83-VM was the "broken" consoleproxy. So the following log starts with
it's expunging.
Afterwards the new consoleproxy shall be deployed - v-85-VM.
This first deployment (logid:dcfe0999) fails due to Insufficent address
capacity - which i can't really understand or see in the database, checking
the availeable IPs for systemVMs (given that i might looking up the wrong
tables....).
How ever CS is working as expected trying to deploy the console proxy
(logid:440fd101) Sync job-4288).
That job fails with "Unable to create deployment, no usable volumes found
for the VM: 85".
Afterwards i have another try to start the consoleproxy (job-4289) which
fails for the same reason. And thats how things go on in the log file...
2022-08-29 17:01:16,224 DEBUG [c.c.v.VirtualMachineManagerImpl]
(API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
Cleaning up hypervisor data structures (ex. SRs in XenServer) for managed
storage. Data from VM instance {id: "83", name: "v-83-VM", uuid:
"b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}.
2022-08-29 17:01:16,225 DEBUG [o.a.c.e.o.VolumeOrchestrator]
(API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
Cleaning storage for vm: 83
2022-08-29 17:01:16,233 DEBUG [c.c.v.VirtualMachineManagerImpl]
(API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
Expunged VM instance {id: "83", name: "v-83-VM", uuid:
"b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"}
2022-08-29 17:01:16,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
Complete async job-4286, jobStatus: SUCCEEDED, resultCode: 0, result:
org.apache.cloudstack.api.response.SystemVmResponse/systemvm/{"id":"b1d1f9c6-eafe-4527-9335-e649c646aab0","systemvmtype":"consoleproxy","zoneid":"48e2e928-3300-43b5-8e3a-d9b358022502","zonename":"xxxxxxxxxx","dns1":"10.129.0.4","dns2":"195.10.195.195","name":"v-83-VM","templateid":"c20799e4-a990-449a-aae6-4b67374c7aff","templatename":"systemvm-kvm-4.17.0","created":"2022-08-29T10:38:26+0200","state":"Stopped","isdynamicallyscalable":"false","hasannotations":"false"}
2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
Publish async job-4286 complete on message bus
2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
Wake up jobs related to job-4286
2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
Update db status for job-4286
2022-08-29 17:01:16,256 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-35:ctx-29e06f28 job-4286 ctx-442b1cb2) (logid:5fe8559a)
Wake up jobs joined with job-4286 and disjoin all subjobs created from job-
4286
2022-08-29 17:01:16,260 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-35:ctx-29e06f28 job-4286) (logid:5fe8559a) Done executing
org.apache.cloudstack.api.command.admin.systemvm.DestroySystemVmCmd for
job-4286
2022-08-29 17:01:16,260 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-35:ctx-29e06f28 job-4286) (logid:5fe8559a) Remove
job-4286 from job monitoring
2022-08-29 17:01:17,691 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-6:ctx-6af18142) (logid:fe263a70) HA health check
task is running...
2022-08-29 17:01:21,694 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-6:ctx-89b1442f) (logid:0efc40b2) HA health check
task is running...
2022-08-29 17:01:25,698 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-5:ctx-62313d05) (logid:7073663f) HA health check
task is running...
2022-08-29 17:01:25,767 DEBUG [c.c.s.StatsCollector]
(secstorage-1:ctx-c80e5a6e) (logid:f8e3eb60) Verifying image storage [1].
Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
2022-08-29 17:01:25,773 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-c80e5a6e) (logid:f8e3eb60) Zone [1] is ready to launch
secondary storage VM.
2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Zone 1 is ready to launch
console proxy
2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Expand console proxy standby
capacity for zone xxxxxxxxxx
2022-08-29 17:01:26,031 DEBUG [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating console proxy
standby capacity for zone [1].
2022-08-29 17:01:26,032 INFO [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) No stopped console proxy is
available, need to allocate a new console proxy
2022-08-29 17:01:26,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Assign console proxy from a
newly started instance for request from data center : 1
2022-08-29 17:01:26,056 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing network
configuration for offering [Network Offering
[1-Public-System-Public-Network]: Network {"id": 200, "name": "null",
"uuid": "1b0bf965-a391-478d-833d-3e37e4a15eb8", "networkofferingid": 1}
2022-08-29 17:01:26,056 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account {"id": 1,
"name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
2022-08-29 17:01:26,064 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing network
configuration for offering [Network Offering
[3-Control-System-Control-Network]: Network {"id": 202, "name": "null",
"uuid": "2af42518-fbf4-4a35-9d6b-370e6ad315c7", "networkofferingid": 3}
2022-08-29 17:01:26,064 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account {"id": 1,
"name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
2022-08-29 17:01:26,069 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Found existing network
configuration for offering [Network Offering
[2-Management-System-Management-Network]: Network {"id": 201, "name":
"null", "uuid": "53711d87-a3c6-4f33-bfce-c30e0de72760",
"networkofferingid": 2}
2022-08-29 17:01:26,069 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Releasing lock for
Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- Account {"id": 1,
"name": "system", "uuid": "60bddbd5-1d8a-11ec-83ce-525400c9c662"}
2022-08-29 17:01:26,077 INFO [c.c.v.VirtualMachineManagerImpl]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) allocating virtual machine
from template:c20799e4-a990-449a-aae6-4b67374c7aff with hostname:v-85-VM
and 3 networks
2022-08-29 17:01:26,079 DEBUG [c.c.v.VirtualMachineManagerImpl]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating entries for VM:
VM instance {id: "85", name: "v-85-VM", uuid:
"110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
2022-08-29 17:01:26,082 DEBUG [c.c.v.VirtualMachineManagerImpl]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating nics for VM
instance {id: "85", name: "v-85-VM", uuid:
"110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}
2022-08-29 17:01:26,083 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Allocating nic for vm VM
instance {id: "85", name: "v-85-VM", uuid:
"110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} in network
Network {"id": 200, "name": "null", "uuid":
"1b0bf965-a391-478d-833d-3e37e4a15eb8", "networkofferingid": 1} with
requested profile NicProfile
{"broadcastUri":null,"iPv4Address":null,"id":0,"reservationId":null,"vmId":0}
2022-08-29 17:01:26,089 WARN [c.c.n.IpAddressManagerImpl]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to get ip address in
zone id=1, vlanId id=[3, 10, 13, 14]
2022-08-29 17:01:26,090 DEBUG [c.c.u.d.T.Transaction]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Rolling back the
transaction: Time = 8 Name = consoleproxy-1; called by
-TransactionLegacy.rollback:888-TransactionLegacy.removeUpTo:831-TransactionLegacy.close:655-Transaction.execute:38-IpAddressManagerImpl.listAvailablePublicIps:824-IpAddressManagerImpl.fetchNewPublicIp:811-IpAddressManagerImpl.fetchNewPublicIp:803-IpAddressManagerImpl.assignPublicIpAddress:784-PublicNetworkGuru.getIp:125-PublicNetworkGuru.allocate:172-NetworkOrchestrator.allocateNic:973-NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice:823
2022-08-29 17:01:26,090 WARN [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to allocate proxy
[Console VM instance {id: "85", name: "v-85-VM", uuid:
"110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
due to [Insufficient address capacity].
com.cloud.exception.InsufficientAddressCapacityException: Insufficient
address capacityScope=interface com.cloud.dc.DataCenter; id=1
at
com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
at
com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
at
com.cloud.network.IpAddressManagerImpl.listAvailablePublicIps(IpAddressManagerImpl.java:824)
at
com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl.java:811)
at
com.cloud.network.IpAddressManagerImpl.fetchNewPublicIp(IpAddressManagerImpl.java:803)
at
com.cloud.network.IpAddressManagerImpl.assignPublicIpAddress(IpAddressManagerImpl.java:784)
at
com.cloud.network.guru.PublicNetworkGuru.getIp(PublicNetworkGuru.java:125)
at
com.cloud.network.guru.PublicNetworkGuru.allocate(PublicNetworkGuru.java:172)
at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocateNic(NetworkOrchestrator.java:973)
at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$3.addRequestedNicToNicListWithDeviceNumberAndRetrieveDefaultDevice(NetworkOrchestrator.java:823)
at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$3.doInTransactionWithoutResult(NetworkOrchestrator.java:797)
at
com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
at
com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.allocate(NetworkOrchestrator.java:778)
at
com.cloud.vm.VirtualMachineManagerImpl$1.doInTransactionWithoutResult(VirtualMachineManagerImpl.java:478)
at
com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
at
com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
at
com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:469)
at
com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:528)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:700)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
at
com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-08-29 17:01:26,091 WARN [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unable to allocate console
proxy standby capacity for zone [1] due to [Unable to allocate proxy
[Console VM instance {id: "85", name: "v-85-VM", uuid:
"110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
due to [Insufficient address capacity].].
com.cloud.utils.exception.CloudRuntimeException: Unable to allocate proxy
[Console VM instance {id: "85", name: "v-85-VM", uuid:
"110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
due to [Insufficient address capacity].
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:704)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
at
com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.exception.InsufficientAddressCapacityException:
Insufficient address capacityScope=interface com.cloud.dc.DataCenter; id=1
at
com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
at
com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
... 38 more
2022-08-29 17:01:26,091 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) received console proxy alert
2022-08-29 17:01:26,092 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Console proxy creation
failure, zone: xxxxxxxxxx
2022-08-29 17:01:26,093 WARN [c.c.a.AlertManagerImpl]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) alertType=[10]
dataCenterId=[1] podId=[null] clusterId=[null] message=[Console proxy
creation failure. zone: xxxxxxxxxx, error details: Unable to allocate proxy
[Console VM instance {id: "85", name: "v-85-VM", uuid:
"110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
due to [Insufficient address capacity].].
2022-08-29 17:01:26,096 WARN [c.c.a.AlertManagerImpl]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) No recipients set in global
setting 'alert.email.addresses', skipping sending alert with subject
[Console proxy creation failure. zone: xxxxxxxxxx, error details: Unable to
allocate proxy [Console VM instance {id: "85", name: "v-85-VM", uuid:
"110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
due to [Insufficient address capacity].] and content [Console proxy
creation failure (zone xxxxxxxxxx)].
2022-08-29 17:01:26,096 WARN [c.c.v.SystemVmLoadScanner]
(consoleproxy-1:ctx-2d236928) (logid:dcfe0999) Unexpected exception Unable
to allocate proxy [Console VM instance {id: "85", name: "v-85-VM", uuid:
"110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
due to [Insufficient address capacity].
com.cloud.utils.exception.CloudRuntimeException: Unable to allocate proxy
[Console VM instance {id: "85", name: "v-85-VM", uuid:
"110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"}] on zone [1]
due to [Insufficient address capacity].
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.createProxyInstance(ConsoleProxyManagerImpl.java:704)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startNew(ConsoleProxyManagerImpl.java:572)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:809)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1537)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:167)
at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:121)
at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:91)
at
com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:82)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.exception.InsufficientAddressCapacityException:
Insufficient address capacityScope=interface com.cloud.dc.DataCenter; id=1
at
com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:950)
at
com.cloud.network.IpAddressManagerImpl$2.doInTransaction(IpAddressManagerImpl.java:824)
at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
... 38 more
2022-08-29 17:01:29,700 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-5:ctx-994465bd) (logid:1e2822fd) HA health check
task is running...
2022-08-29 17:01:30,444 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 2 routers to
update status.
2022-08-29 17:01:30,449 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 1 VPC's to
update Redundant State.
2022-08-29 17:01:30,451 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-e502832a) (logid:9f124ba7) Found 0 networks to
update RvR status.
2022-08-29 17:01:30,459 DEBUG [c.c.a.t.Request]
(RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
6-4833769775052102463: Sending { Cmd , MgmtId: 90520740759984, via:
6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.name":"r-79-VM","router.ip":"169.254.144.198"},"wait":"30","bypassHostMaintenance":"false"}}]
}
2022-08-29 17:01:30,643 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-12:null) (logid:) Seq 6-4833769775052102463:
Processing: { Ans: , MgmtId: 90520740759984, via: 6, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.CheckRouterAnswer":{"state":"PRIMARY","result":"true","details":"Status:
PRIMARY
","wait":"0","bypassHostMaintenance":"false"}}] }
2022-08-29 17:01:30,643 DEBUG [c.c.a.t.Request]
(RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
6-4833769775052102463: Received: { Ans: , MgmtId: 90520740759984, via:
6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer } }
2022-08-29 17:01:30,643 DEBUG [c.c.a.m.AgentManagerImpl]
(RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Details from
executing class com.cloud.agent.api.CheckRouterCommand: Status: PRIMARY
2022-08-29 17:01:30,655 DEBUG [c.c.a.t.Request]
(RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
1-5410793477308947401: Sending { Cmd , MgmtId: 90520740759984, via:
1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.name":"r-80-VM","router.ip":"169.254.217.200"},"wait":"30","bypassHostMaintenance":"false"}}]
}
2022-08-29 17:01:30,796 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-7:null) (logid:) Seq 1-5410793477308947401:
Processing: { Ans: , MgmtId: 90520740759984, via: 1, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.CheckRouterAnswer":{"state":"BACKUP","result":"true","details":"Status:
BACKUP
","wait":"0","bypassHostMaintenance":"false"}}] }
2022-08-29 17:01:30,796 DEBUG [c.c.a.t.Request]
(RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Seq
1-5410793477308947401: Received: { Ans: , MgmtId: 90520740759984, via:
1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer } }
2022-08-29 17:01:30,796 DEBUG [c.c.a.m.AgentManagerImpl]
(RedundantRouterStatusMonitor-7:ctx-e3616662) (logid:2b499e08) Details from
executing class com.cloud.agent.api.CheckRouterCommand: Status: BACKUP
2022-08-29 17:01:33,703 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-5:ctx-70bca202) (logid:b332dbe0) HA health check
task is running...
2022-08-29 17:01:34,083 DEBUG [c.c.s.StatsCollector]
(StatsCollector-6:ctx-213e2ecc) (logid:aa0584c9) AutoScaling Monitor is
running...
2022-08-29 17:01:37,707 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-6:ctx-57493517) (logid:5093e75f) HA health check
task is running...
2022-08-29 17:01:38,640 DEBUG [c.c.s.StatsCollector]
(StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) HostStatsCollector is
running...
2022-08-29 17:01:38,655 DEBUG [c.c.a.t.Request]
(StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq 1-5410793477308947402:
Received: { Ans: , MgmtId: 90520740759984, via:
1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2022-08-29 17:01:38,707 DEBUG [c.c.a.t.Request]
(StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq 5-8959067033723473395:
Received: { Ans: , MgmtId: 90520740759984, via:
5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2022-08-29 17:01:38,717 DEBUG [c.c.a.t.Request]
(StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq 6-4833769775052102464:
Received: { Ans: , MgmtId: 90520740759984, via:
6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2022-08-29 17:01:38,725 DEBUG [c.c.a.t.Request]
(StatsCollector-3:ctx-b503d899) (logid:bcbb4f5c) Seq 7-295548725546192472:
Received: { Ans: , MgmtId: 90520740759984, via:
7(srv-5-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2022-08-29 17:01:41,709 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-5:ctx-a65ae462) (logid:bc2b7320) HA health check
task is running...
2022-08-29 17:01:45,712 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-2:ctx-05491ace) (logid:b2232c4d) HA health check
task is running...
2022-08-29 17:01:47,019 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-14:null) (logid:) Ping from Routing host
7(srv-5-comp.aaaaaaaa.bbb)
2022-08-29 17:01:49,715 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-2:ctx-7943581c) (logid:504c98ff) HA health check
task is running...
2022-08-29 17:01:51,881 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-13:null) (logid:) Ping from Routing host
1(srv-2-comp.aaaaaaaa.bbb)
2022-08-29 17:01:53,718 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-4:ctx-b11cfa22) (logid:3163613f) HA health check
task is running...
2022-08-29 17:01:53,955 DEBUG [c.c.a.t.Request]
(StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq 1-5410793477308947403:
Received: { Ans: , MgmtId: 90520740759984, via:
1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
2022-08-29 17:01:54,025 DEBUG [c.c.a.t.Request]
(StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq 5-8959067033723473396:
Received: { Ans: , MgmtId: 90520740759984, via:
5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
2022-08-29 17:01:54,096 DEBUG [c.c.a.t.Request]
(StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq 6-4833769775052102465:
Received: { Ans: , MgmtId: 90520740759984, via:
6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
2022-08-29 17:01:54,217 DEBUG [c.c.a.t.Request]
(StatsCollector-4:ctx-c883ff8a) (logid:071b02e7) Seq 7-295548725546192473:
Received: { Ans: , MgmtId: 90520740759984, via:
7(srv-5-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
2022-08-29 17:01:55,765 DEBUG [c.c.s.StatsCollector]
(secstorage-1:ctx-7d9995d7) (logid:e388a8b8) Verifying image storage [1].
Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
2022-08-29 17:01:55,771 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-7d9995d7) (logid:e388a8b8) Zone [1] is ready to launch
secondary storage VM.
2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-413f712d) (logid:440fd101) Zone 1 is ready to launch
console proxy
2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-413f712d) (logid:440fd101) Expand console proxy standby
capacity for zone xxxxxxxxxx
2022-08-29 17:01:56,033 DEBUG [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-413f712d) (logid:440fd101) Allocating console proxy
standby capacity for zone [1].
2022-08-29 17:01:56,034 INFO [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-413f712d) (logid:440fd101) Found a stopped console
proxy, starting it. Vm id : 85
2022-08-29 17:01:56,035 DEBUG [c.c.v.VirtualMachineManagerImpl]
(consoleproxy-1:ctx-413f712d) (logid:440fd101) start parameter value of
enterHardwareSetup == <very null> during processing of queued job
2022-08-29 17:01:56,044 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(consoleproxy-1:ctx-413f712d) (logid:440fd101) Sync job-4288 execution on
object VmWorkJobQueue.85
2022-08-29 17:01:56,173 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-e75826b1) (logid:e80fe659) Execute sync-queue
item: SyncQueueItemVO {id:2105, queueId: 2105, contentType: AsyncJob,
contentId: 4288, lastProcessMsid: 90520740759984, lastprocessNumber: 1,
lastProcessTime: Mon Aug 29 17:01:56 CEST 2022, created: Mon Aug 29
17:01:56 CEST 2022}
2022-08-29 17:01:56,174 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-e75826b1) (logid:e80fe659) Schedule queued
job-4288
2022-08-29 17:01:56,179 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:7e671dd0) Add
job-4288 into job monitoring
2022-08-29 17:01:56,186 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5)
Executing AsyncJobVO: {id:4288, userId: 1, accountId: 1, instanceType:
null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAVXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 90520740759984, completeMsid: null, lastUpdated:
null, lastPolled: null, created: Mon Aug 29 17:01:56 CEST 2022, removed:
null}
2022-08-29 17:01:56,186 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5) Run
VM work job: com.cloud.vm.VmWorkStart for VM 85, job origin: 3445
2022-08-29 17:01:56,187 DEBUG [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
(logid:ba10fec5) Execute VM work job:
com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
2022-08-29 17:01:56,189 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
(logid:ba10fec5) orchestrating VM start for 'v-85-VM'
com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
2022-08-29 17:01:56,196 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
(logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
"110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
transited from [Stopped] to [Starting] with event [StartRequested]. VM's
original host: null, new host: null, host before state transition: null
2022-08-29 17:01:56,196 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
(logid:ba10fec5) Successfully transitioned to start state for VM instance
{id: "85", name: "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
type="ConsoleProxy"} reservation id = 0ab68202-d8be-469c-9b2a-87a55c2dc489
2022-08-29 17:01:56,200 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
(logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts: null
2022-08-29 17:01:56,202 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
(logid:ba10fec5) VM start attempt #1
2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
(logid:ba10fec5) DeploymentPlanner allocation algorithm: null
2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
(logid:ba10fec5) Trying to allocate a host and storage pools from dc:1,
pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
1073741824
2022-08-29 17:01:56,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
(logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
2022-08-29 17:01:56,212 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
(logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
2022-08-29 17:01:56,214 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
(logid:ba10fec5) Searching all possible resources under this Zone: 1
2022-08-29 17:01:56,215 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
(logid:ba10fec5) Listing clusters in order of aggregate capacity, that have
(at least one host with) enough CPU and RAM capacity under this Zone: 1
2022-08-29 17:01:56,218 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
(logid:ba10fec5) Removing from the clusterId list these clusters from avoid
set: []
2022-08-29 17:01:56,224 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
(logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
2022-08-29 17:01:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested with
Custom[UEFI] Boot Type false
2022-08-29 17:01:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in dc: 1
pod:1 cluster:1
2022-08-29 17:01:56,226 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4 hosts to
check for allocation: [Host {"id": "5", "name": "srv-4-comp.aaaaaaaa.bbb",
"uuid": "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
{"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb", "uuid":
"8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host {"id": "6",
"name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
"6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id": "1",
"name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
"883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
2022-08-29 17:01:56,232 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for allocation
after prioritization: [Host {"id": "5", "name": "srv-4-comp.aaaaaaaa.bbb",
"uuid": "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host
{"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb", "uuid":
"8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host {"id": "6",
"name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
"6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id": "1",
"name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
"883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
2022-08-29 17:01:56,232 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Looking for speed=500Mhz,
Ram=1024 MB
2022-08-29 17:01:56,232 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
srv-4-comp.aaaaaaaa.bbb, uuid: 134c625d-15ab-461a-a38d-36fb0b979b5d} is KVM
hypervisor type, no max guest limit check needed
2022-08-29 17:01:56,234 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu capability
(cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
2022-08-29 17:01:56,234 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has enough
capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
cpuOverprovisioningFactor: 1.0
2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
and CPU after applying overprovisioning: 127200
2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 , Requested
CPU: 500
2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
251009040384 , Requested RAM: (1.00 GB) 1073741824
2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
available
2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
5, used: 11100, reserved: 0, actual total: 127200, total with
overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
,considerReservedCapacity?: true
2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total: (250.27 GB)
268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
false , considerReservedCapacity?: true
2022-08-29 17:01:56,235 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
list: 5
2022-08-29 17:01:56,235 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
srv-5-comp.aaaaaaaa.bbb, uuid: 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is KVM
hypervisor type, no max guest limit check needed
2022-08-29 17:01:56,238 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu capability
(cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
2022-08-29 17:01:56,238 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has enough
capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
cpuOverprovisioningFactor: 1.0
2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
and CPU after applying overprovisioning: 127200
2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 , Requested CPU:
500
2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
189373743104 , Requested RAM: (1.00 GB) 1073741824
2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
available
2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
7, used: 90100, reserved: 0, actual total: 127200, total with
overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
,considerReservedCapacity?: true
2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total: (250.27 GB)
268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
false , considerReservedCapacity?: true
2022-08-29 17:01:56,240 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
list: 7
2022-08-29 17:01:56,240 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
srv-3-comp.aaaaaaaa.bbb, uuid: 6858713b-7e17-46d8-854d-24a13f9c7001} is KVM
hypervisor type, no max guest limit check needed
2022-08-29 17:01:56,242 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu capability
(cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
2022-08-29 17:01:56,242 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has enough
capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
cpuOverprovisioningFactor: 1.0
2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
and CPU after applying overprovisioning: 127200
2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 , Requested CPU:
500
2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
204032835584 , Requested RAM: (1.00 GB) 1073741824
2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
available
2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
6, used: 42900, reserved: 0, actual total: 127200, total with
overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
,considerReservedCapacity?: true
2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total: (250.27 GB)
268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
false , considerReservedCapacity?: true
2022-08-29 17:01:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
list: 6
2022-08-29 17:01:56,244 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
srv-2-comp.aaaaaaaa.bbb, uuid: 883f7f62-55f2-42f4-8242-16b6847c4204} is KVM
hypervisor type, no max guest limit check needed
2022-08-29 17:01:56,246 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu capability
(cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
2022-08-29 17:01:56,246 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has enough
capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
cpuOverprovisioningFactor: 1.0
2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
and CPU after applying overprovisioning: 127200
2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 , Requested CPU:
500
2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
229802639360 , Requested RAM: (1.00 GB) 1073741824
2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
available
2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
1, used: 48200, reserved: 0, actual total: 127200, total with
overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
,considerReservedCapacity?: true
2022-08-29 17:01:56,247 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total: (250.27 GB)
268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
false , considerReservedCapacity?: true
2022-08-29 17:01:56,247 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
list: 1
2022-08-29 17:01:56,247 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a
FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator returning 4
suitable hosts
2022-08-29 17:01:56,258 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
(logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
"110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
transited from [Starting] to [Stopped] with event [OperationFailed]. VM's
original host: null, new host: null, host before state transition: null
2022-08-29 17:01:56,258 WARN [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
(logid:ba10fec5) Unable to orchestrate start VM instance {id: "85", name:
"v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
type="ConsoleProxy"} due to [Unable to create deployment, no usable volumes
found for the VM: 85].
com.cloud.utils.exception.CloudRuntimeException: Unable to create
deployment, no usable volumes found for the VM: 85
at
com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
at
com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
at
com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-08-29 17:01:56,259 DEBUG [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
(logid:ba10fec5) Done executing VM work job:
com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
2022-08-29 17:01:56,260 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
(logid:ba10fec5) Complete async job-4288, jobStatus: FAILED, resultCode: 0,
result:
rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODUiLCBuYW1lOiAidi04NS1WTSIsIHV1aWQ6ICIxMTBkODYyMC1jMDI5LTRjYTktOTA4NC0yMTAwOTk3MjVhODAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODVdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
(logid:ba10fec5) Publish async job-4288 complete on message bus
2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
(logid:ba10fec5) Wake up jobs related to job-4288
2022-08-29 17:01:56,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
(logid:ba10fec5) Update db status for job-4288
2022-08-29 17:01:56,262 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288 ctx-2852a95a)
(logid:ba10fec5) Wake up jobs joined with job-4288 and disjoin all subjobs
created from job- 4288
2022-08-29 17:01:56,270 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5)
Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 85, job
origin: 3445
2022-08-29 17:01:56,270 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5)
Done executing com.cloud.vm.VmWorkStart for job-4288
2022-08-29 17:01:56,272 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-139:ctx-26560f3a job-3445/job-4288) (logid:ba10fec5)
Remove job-4288 from job monitoring
2022-08-29 17:01:56,277 WARN [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-413f712d) (logid:440fd101) Unable to allocate console
proxy standby capacity for zone [1] due to [Unable to orchestrate start VM
instance {id: "85", name: "v-85-VM", uuid:
"110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to [Unable
to create deployment, no usable volumes found for the VM: 85].].
com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate
start VM instance {id: "85", name: "v-85-VM", uuid:
"110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to [Unable
to create deployment, no usable volumes found for the VM: 85].
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-08-29 17:01:56,277 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
(consoleproxy-1:ctx-413f712d) (logid:440fd101) received console proxy alert
2022-08-29 17:01:56,278 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
(consoleproxy-1:ctx-413f712d) (logid:440fd101) Console proxy creation
failure, zone: xxxxxxxxxx
2022-08-29 17:01:56,279 WARN [c.c.a.AlertManagerImpl]
(consoleproxy-1:ctx-413f712d) (logid:440fd101) alertType=[10]
dataCenterId=[1] podId=[null] clusterId=[null] message=[Console proxy
creation failure. zone: xxxxxxxxxx, error details: Unable to orchestrate
start VM instance {id: "85", name: "v-85-VM", uuid:
"110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to [Unable
to create deployment, no usable volumes found for the VM: 85].].
2022-08-29 17:01:56,284 WARN [c.c.a.AlertManagerImpl]
(consoleproxy-1:ctx-413f712d) (logid:440fd101) No recipients set in global
setting 'alert.email.addresses', skipping sending alert with subject
[Console proxy creation failure. zone: xxxxxxxxxx, error details: Unable to
orchestrate start VM instance {id: "85", name: "v-85-VM", uuid:
"110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to [Unable
to create deployment, no usable volumes found for the VM: 85].] and content
[Console proxy creation failure (zone xxxxxxxxxx)].
2022-08-29 17:01:56,284 WARN [c.c.v.SystemVmLoadScanner]
(consoleproxy-1:ctx-413f712d) (logid:440fd101) Unexpected exception Unable
to orchestrate start VM instance {id: "85", name: "v-85-VM", uuid:
"110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to [Unable
to create deployment, no usable volumes found for the VM: 85].
com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate
start VM instance {id: "85", name: "v-85-VM", uuid:
"110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to [Unable
to create deployment, no usable volumes found for the VM: 85].
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-08-29 17:01:57,721 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-4:ctx-9528226f) (logid:4512ff9b) HA health check
task is running...
2022-08-29 17:01:59,504 DEBUG [c.c.s.StatsCollector]
(StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) StorageCollector is
running...
2022-08-29 17:01:59,518 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
(StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) getCommandHostDelegation:
class com.cloud.agent.api.GetStorageStatsCommand
2022-08-29 17:01:59,518 DEBUG [c.c.h.XenServerGuru]
(StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning the
default host to execute commands because the command is not of Copy type.
2022-08-29 17:01:59,571 DEBUG [c.c.a.t.Request]
(StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq
11-7523826127475836583: Received: { Ans: , MgmtId: 90520740759984, via:
11(s-32-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2022-08-29 17:01:59,574 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
(StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) getCommandHostDelegation:
class com.cloud.agent.api.GetStorageStatsCommand
2022-08-29 17:01:59,574 DEBUG [c.c.h.XenServerGuru]
(StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning the
default host to execute commands because the command is not of Copy type.
2022-08-29 17:01:59,615 DEBUG [c.c.a.t.Request]
(StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq 5-8959067033723473397:
Received: { Ans: , MgmtId: 90520740759984, via:
5(srv-4-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2022-08-29 17:01:59,624 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
(StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) getCommandHostDelegation:
class com.cloud.agent.api.GetStorageStatsCommand
2022-08-29 17:01:59,624 DEBUG [c.c.h.XenServerGuru]
(StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) We are returning the
default host to execute commands because the command is not of Copy type.
2022-08-29 17:01:59,653 DEBUG [c.c.a.t.Request]
(StatsCollector-2:ctx-67cb5761) (logid:0cd67bc8) Seq 6-4833769775052102466:
Received: { Ans: , MgmtId: 90520740759984, via:
6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
2022-08-29 17:01:59,979 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-12:null) (logid:) Ping from Routing host
5(srv-4-comp.aaaaaaaa.bbb)
2022-08-29 17:02:00,269 DEBUG [c.c.s.StatsCollector]
(StatsCollector-5:ctx-d2b6ffd3) (logid:95201678) DbCollector is running...
2022-08-29 17:02:00,443 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 2 routers to
update status.
2022-08-29 17:02:00,449 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 1 VPC's to
update Redundant State.
2022-08-29 17:02:00,450 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-10041d6f) (logid:9c8fd7f4) Found 0 networks to
update RvR status.
2022-08-29 17:02:00,457 DEBUG [c.c.a.t.Request]
(RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
6-4833769775052102467: Sending { Cmd , MgmtId: 90520740759984, via:
6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.name":"r-79-VM","router.ip":"169.254.144.198"},"wait":"30","bypassHostMaintenance":"false"}}]
}
2022-08-29 17:02:00,599 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-7:null) (logid:) Seq 6-4833769775052102467:
Processing: { Ans: , MgmtId: 90520740759984, via: 6, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.CheckRouterAnswer":{"state":"PRIMARY","result":"true","details":"Status:
PRIMARY
","wait":"0","bypassHostMaintenance":"false"}}] }
2022-08-29 17:02:00,599 DEBUG [c.c.a.t.Request]
(RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
6-4833769775052102467: Received: { Ans: , MgmtId: 90520740759984, via:
6(srv-3-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer } }
2022-08-29 17:02:00,599 DEBUG [c.c.a.m.AgentManagerImpl]
(RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Details
from executing class com.cloud.agent.api.CheckRouterCommand: Status: PRIMARY
2022-08-29 17:02:00,612 DEBUG [c.c.a.t.Request]
(RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
1-5410793477308947404: Sending { Cmd , MgmtId: 90520740759984, via:
1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.CheckRouterCommand":{"accessDetails":{"router.name":"r-80-VM","router.ip":"169.254.217.200"},"wait":"30","bypassHostMaintenance":"false"}}]
}
2022-08-29 17:02:00,748 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-4:null) (logid:) Ping from 11(s-32-VM)
2022-08-29 17:02:00,750 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-1:null) (logid:) Seq 1-5410793477308947404:
Processing: { Ans: , MgmtId: 90520740759984, via: 1, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.CheckRouterAnswer":{"state":"BACKUP","result":"true","details":"Status:
BACKUP
","wait":"0","bypassHostMaintenance":"false"}}] }
2022-08-29 17:02:00,750 DEBUG [c.c.a.t.Request]
(RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Seq
1-5410793477308947404: Received: { Ans: , MgmtId: 90520740759984, via:
1(srv-2-comp.aaaaaaaa.bbb), Ver: v1, Flags: 10, { CheckRouterAnswer } }
2022-08-29 17:02:00,751 DEBUG [c.c.a.m.AgentManagerImpl]
(RedundantRouterStatusMonitor-10:ctx-09a24dce) (logid:da20227f) Details
from executing class com.cloud.agent.api.CheckRouterCommand: Status: BACKUP
2022-08-29 17:02:01,362 DEBUG [c.c.s.StatsCollector]
(Cluster-Worker-839:ctx-e097a8ab) (logid:3deacf65) StatusUpdate from
90520740759984, json:
{"managementServerHostId":1,"managementServerHostUuid":"5615ca5f-0716-4da1-93ec-31f79fd19bbe","collectionTime":"Aug
29, 2022, 5:02:01
PM","sessions":1,"cpuUtilization":0.0,"totalJvmMemoryBytes":324534272,"freeJvmMemoryBytes":68595424,"maxJvmMemoryBytes":1908932607,"processJvmMemoryBytes":0,"jvmUptime":103760388,"jvmStartTime":1661681560910,"availableProcessors":6,"loadAverage":0.02,"totalInit":139788288,"totalUsed":474791104,"totalCommitted":557662208,"pid":1522332,"jvmName":"1522332@vm-1-cs-mgmt.aaaaaaaa.bbb","jvmVendor":"Ubuntu","jvmVersion":"11.0.16+8-post-Ubuntu-0ubuntu120.04","osDistribution":"Ubuntu
20.04.5
LTS","agentCount":5,"heapMemoryUsed":256775312,"heapMemoryTotal":1908932608,"threadsBlockedCount":0,"threadsDaemonCount":13,"threadsRunnableCount":15,"threadsTerminatedCount":0,"threadsTotalCount":420,"threadsWaitingCount":331,"systemMemoryTotal":8343236608,"systemMemoryFree":2623193088,"systemMemoryUsed":814808,"systemMemoryVirtualSize":7494184960,"logInfo":"/var/log/cloudstack/management/management-server.log
using: 74M\non disk /dev/vda2 mounted on / (14%
full)","systemTotalCpuCycles":15870.180000000002,"systemLoadAverages":[0.02,0.03,0.0],"systemCyclesUsage":[4116569,2615101,2716460424],"dbLocal":false,"usageLocal":false,"systemBootTime":"Jul
8, 2022, 2:35:55 AM","kernelVersion":"5.4.0-121-generic"}
2022-08-29 17:02:01,723 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-4:ctx-7abb1b5d) (logid:2320950d) HA health check
task is running...
2022-08-29 17:02:05,417 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Vm-Operations-Cleanup-1:ctx-e741e208) (logid:3f5129e0) VM Operation Thread
Running
2022-08-29 17:02:05,725 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-4:ctx-8e7f6766) (logid:710cb3b2) HA health check
task is running...
2022-08-29 17:02:08,626 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-11:null) (logid:) Ping from Routing host
6(srv-3-comp.aaaaaaaa.bbb)
2022-08-29 17:02:09,727 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-4:ctx-26a5ff07) (logid:09b3c89a) HA health check
task is running...
2022-08-29 17:02:13,729 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-4:ctx-b4d4e374) (logid:9048c7a7) HA health check
task is running...
2022-08-29 17:02:17,731 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-4:ctx-3db3d208) (logid:7bb73f7b) HA health check
task is running...
2022-08-29 17:02:21,734 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-1:ctx-fca1d248) (logid:1ed5381a) HA health check
task is running...
2022-08-29 17:02:25,737 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-1:ctx-984e2382) (logid:e6c3d724) HA health check
task is running...
2022-08-29 17:02:25,759 DEBUG [c.c.s.StatsCollector]
(secstorage-1:ctx-64c610ed) (logid:6603cd38) Verifying image storage [1].
Capacity: total=[1 TB], used=[828 GB], threshold=[100.0%].
2022-08-29 17:02:25,763 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-64c610ed) (logid:6603cd38) Zone [1] is ready to launch
secondary storage VM.
2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Zone 1 is ready to launch
console proxy
2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Expand console proxy standby
capacity for zone xxxxxxxxxx
2022-08-29 17:02:26,029 DEBUG [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Allocating console proxy
standby capacity for zone [1].
2022-08-29 17:02:26,030 INFO [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Found a stopped console
proxy, starting it. Vm id : 85
2022-08-29 17:02:26,031 DEBUG [c.c.v.VirtualMachineManagerImpl]
(consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) start parameter value of
enterHardwareSetup == <very null> during processing of queued job
2022-08-29 17:02:26,037 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Sync job-4289 execution on
object VmWorkJobQueue.85
2022-08-29 17:02:26,167 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-bba6b89d) (logid:f9aed0eb) Execute sync-queue
item: SyncQueueItemVO {id:2106, queueId: 2105, contentType: AsyncJob,
contentId: 4289, lastProcessMsid: 90520740759984, lastprocessNumber: 2,
lastProcessTime: Mon Aug 29 17:02:26 CEST 2022, created: Mon Aug 29
17:02:26 CEST 2022}
2022-08-29 17:02:26,167 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-bba6b89d) (logid:f9aed0eb) Schedule queued
job-4289
2022-08-29 17:02:26,172 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:f919f9b3) Add
job-4289 into job monitoring
2022-08-29 17:02:26,181 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5)
Executing AsyncJobVO: {id:4289, userId: 1, accountId: 1, instanceType:
null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAVXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwcHA,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 90520740759984, completeMsid: null, lastUpdated:
null, lastPolled: null, created: Mon Aug 29 17:02:26 CEST 2022, removed:
null}
2022-08-29 17:02:26,181 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5) Run
VM work job: com.cloud.vm.VmWorkStart for VM 85, job origin: 3445
2022-08-29 17:02:26,183 DEBUG [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
(logid:ba10fec5) Execute VM work job:
com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
2022-08-29 17:02:26,184 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
(logid:ba10fec5) orchestrating VM start for 'v-85-VM'
com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
2022-08-29 17:02:26,192 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
(logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
"110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
transited from [Stopped] to [Starting] with event [StartRequested]. VM's
original host: null, new host: null, host before state transition: null
2022-08-29 17:02:26,192 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
(logid:ba10fec5) Successfully transitioned to start state for VM instance
{id: "85", name: "v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
type="ConsoleProxy"} reservation id = 24bf955f-96d8-4f4f-ae33-2cae2f3e0425
2022-08-29 17:02:26,198 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
(logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts: null
2022-08-29 17:02:26,200 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
(logid:ba10fec5) VM start attempt #1
2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
(logid:ba10fec5) DeploymentPlanner allocation algorithm: null
2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
(logid:ba10fec5) Trying to allocate a host and storage pools from dc:1,
pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
1073741824
2022-08-29 17:02:26,203 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
(logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
2022-08-29 17:02:26,211 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
(logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
2022-08-29 17:02:26,212 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
(logid:ba10fec5) Searching all possible resources under this Zone: 1
2022-08-29 17:02:26,213 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
(logid:ba10fec5) Listing clusters in order of aggregate capacity, that have
(at least one host with) enough CPU and RAM capacity under this Zone: 1
2022-08-29 17:02:26,216 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
(logid:ba10fec5) Removing from the clusterId list these clusters from avoid
set: []
2022-08-29 17:02:26,221 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
(logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
2022-08-29 17:02:26,222 INFO [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested with
Custom[UEFI] Boot Type false
2022-08-29 17:02:26,223 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in dc: 1
pod:1 cluster:1
2022-08-29 17:02:26,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4 hosts to
check for allocation: [Host {"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb",
"uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
{"id": "1", "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
"883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}, Host {"id": "6",
"name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
"6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id": "5",
"name": "srv-4-comp.aaaaaaaa.bbb", "uuid":
"134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}]
2022-08-29 17:02:26,231 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for allocation
after prioritization: [Host {"id": "7", "name": "srv-5-comp.aaaaaaaa.bbb",
"uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
{"id": "1", "name": "srv-2-comp.aaaaaaaa.bbb", "uuid":
"883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}, Host {"id": "6",
"name": "srv-3-comp.aaaaaaaa.bbb", "uuid":
"6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id": "5",
"name": "srv-4-comp.aaaaaaaa.bbb", "uuid":
"134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}]
2022-08-29 17:02:26,231 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Looking for speed=500Mhz,
Ram=1024 MB
2022-08-29 17:02:26,231 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
srv-5-comp.aaaaaaaa.bbb, uuid: 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is KVM
hypervisor type, no max guest limit check needed
2022-08-29 17:02:26,234 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu capability
(cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
2022-08-29 17:02:26,234 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has enough
capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
cpuOverprovisioningFactor: 1.0
2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
and CPU after applying overprovisioning: 127200
2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 , Requested CPU:
500
2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
189373743104 , Requested RAM: (1.00 GB) 1073741824
2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
available
2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
7, used: 90100, reserved: 0, actual total: 127200, total with
overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
,considerReservedCapacity?: true
2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total: (250.27 GB)
268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
false , considerReservedCapacity?: true
2022-08-29 17:02:26,236 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
list: 7
2022-08-29 17:02:26,236 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
srv-2-comp.aaaaaaaa.bbb, uuid: 883f7f62-55f2-42f4-8242-16b6847c4204} is KVM
hypervisor type, no max guest limit check needed
2022-08-29 17:02:26,239 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu capability
(cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
2022-08-29 17:02:26,239 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has enough
capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
cpuOverprovisioningFactor: 1.0
2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
and CPU after applying overprovisioning: 127200
2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 , Requested CPU:
500
2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
229802639360 , Requested RAM: (1.00 GB) 1073741824
2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
available
2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
1, used: 48200, reserved: 0, actual total: 127200, total with
overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
,considerReservedCapacity?: true
2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total: (250.27 GB)
268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
false , considerReservedCapacity?: true
2022-08-29 17:02:26,240 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
list: 1
2022-08-29 17:02:26,240 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
srv-3-comp.aaaaaaaa.bbb, uuid: 6858713b-7e17-46d8-854d-24a13f9c7001} is KVM
hypervisor type, no max guest limit check needed
2022-08-29 17:02:26,243 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu capability
(cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
2022-08-29 17:02:26,243 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has enough
capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
cpuOverprovisioningFactor: 1.0
2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
and CPU after applying overprovisioning: 127200
2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 , Requested CPU:
500
2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
204032835584 , Requested RAM: (1.00 GB) 1073741824
2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
available
2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
6, used: 42900, reserved: 0, actual total: 127200, total with
overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
,considerReservedCapacity?: true
2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total: (250.27 GB)
268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
false , considerReservedCapacity?: true
2022-08-29 17:02:26,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
list: 6
2022-08-29 17:02:26,244 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
srv-4-comp.aaaaaaaa.bbb, uuid: 134c625d-15ab-461a-a38d-36fb0b979b5d} is KVM
hypervisor type, no max guest limit check needed
2022-08-29 17:02:26,247 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu capability
(cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
2022-08-29 17:02:26,247 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has enough
capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
cpuOverprovisioningFactor: 1.0
2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
and CPU after applying overprovisioning: 127200
2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 , Requested
CPU: 500
2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
251009040384 , Requested RAM: (1.00 GB) 1073741824
2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
available
2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
5, used: 11100, reserved: 0, actual total: 127200, total with
overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
,considerReservedCapacity?: true
2022-08-29 17:02:26,249 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total: (250.27 GB)
268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
false , considerReservedCapacity?: true
2022-08-29 17:02:26,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
list: 5
2022-08-29 17:02:26,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb
FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator returning 4
suitable hosts
2022-08-29 17:02:26,260 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
(logid:ba10fec5) VM instance {id: "85", name: "v-85-VM", uuid:
"110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} state
transited from [Starting] to [Stopped] with event [OperationFailed]. VM's
original host: null, new host: null, host before state transition: null
2022-08-29 17:02:26,260 WARN [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
(logid:ba10fec5) Unable to orchestrate start VM instance {id: "85", name:
"v-85-VM", uuid: "110d8620-c029-4ca9-9084-210099725a80",
type="ConsoleProxy"} due to [Unable to create deployment, no usable volumes
found for the VM: 85].
com.cloud.utils.exception.CloudRuntimeException: Unable to create
deployment, no usable volumes found for the VM: 85
at
com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
at
com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
at
com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-08-29 17:02:26,261 DEBUG [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
(logid:ba10fec5) Done executing VM work job:
com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":85,"handlerName":"VirtualMachineManagerImpl"}
2022-08-29 17:02:26,262 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
(logid:ba10fec5) Complete async job-4289, jobStatus: FAILED, resultCode: 0,
result:
rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODUiLCBuYW1lOiAidi04NS1WTSIsIHV1aWQ6ICIxMTBkODYyMC1jMDI5LTRjYTktOTA4NC0yMTAwOTk3MjVhODAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODVdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
(logid:ba10fec5) Publish async job-4289 complete on message bus
2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
(logid:ba10fec5) Wake up jobs related to job-4289
2022-08-29 17:02:26,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
(logid:ba10fec5) Update db status for job-4289
2022-08-29 17:02:26,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289 ctx-214097cb)
(logid:ba10fec5) Wake up jobs joined with job-4289 and disjoin all subjobs
created from job- 4289
2022-08-29 17:02:26,269 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5)
Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 85, job
origin: 3445
2022-08-29 17:02:26,270 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5)
Done executing com.cloud.vm.VmWorkStart for job-4289
2022-08-29 17:02:26,271 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-140:ctx-67320e13 job-3445/job-4289) (logid:ba10fec5)
Remove job-4289 from job monitoring
2022-08-29 17:02:26,277 WARN [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) Unable to allocate console
proxy standby capacity for zone [1] due to [Unable to orchestrate start VM
instance {id: "85", name: "v-85-VM", uuid:
"110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to [Unable
to create deployment, no usable volumes found for the VM: 85].].
com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate
start VM instance {id: "85", name: "v-85-VM", uuid:
"110d8620-c029-4ca9-9084-210099725a80", type="ConsoleProxy"} due to [Unable
to create deployment, no usable volumes found for the VM: 85].
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-08-29 17:02:26,277 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
(consoleproxy-1:ctx-0d1bbbb1) (logid:452c90ec) received console proxy alert
Am Mo., 29. Aug. 2022 um 16:30 Uhr schrieb Slavka Peleva <
slavkap@storpool.com>:
> Thanks, Chris, for the information!
>
> Are you using custom offerings for the system VMs? Can you share more info
> if yes?
>
> About the log, it's not the complete log from where the deployment
> started. The deployment begins a bit earlier than what you shared. Can you
> search the management log for `Sync job-3539 execution on object
> VmWorkJobQueue` to get the `logid` for the volume creation (coloured in
> blue)? And if possible, share the logs with the found log ID
>
> Here is an example from my logs:
>
> 2022-08-29 17:07:49,114 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (consoleproxy-1:ctx-a952a3dc) (logid:89157bb9) Sync job-60484 execution
> on object VmWorkJobQueue.4904
>
> 2022-08-29 17:07:50,816 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-5:ctx-56172581 job-60479/job-60484 ctx-a900986d)
> (logid:95d750ae) Execute VM work job:
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":4904,"handlerName":"VirtualMachineManagerImpl"}
> 2022-08-29 17:07:50,817 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-5:ctx-56172581 job-60479/job-60484 ctx-a900986d)
> (logid:95d750ae) orchestrating VM start for 'v-4904-VM'
> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
>
>
> On Mon, Aug 29, 2022 at 2:29 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
>
>> Hi Slavka,
>>
>> thats what i did after your suggestion / informatin and where the error
>> occured.
>> Deleted / Destroyed the consoleproxy and waited till recreation. Which
>> didn't worked throwing the error i pasted above.
>> Sorry for beeing so unspecific.
>>
>> Regards,
>> Chris
>>
>> Am Mo., 29. Aug. 2022 um 12:46 Uhr schrieb Slavka Peleva <
>> slavkap@storpool.com>:
>>
>>> Hi Chris,
>>>
>>> I mean to recreate the system VMs by destroying them.
>>>
>>> On Mon, Aug 29, 2022 at 12:38 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
>>>
>>>> Hi Slavka,
>>>>
>>>> didn't tried to restart the SystemVMs. Reading through the docs i had
>>>> the impression that "live-patching" would be enough for this....
>>>> Anyway - I tried my luck with the console proxy.
>>>>
>>>> Which failed utterly. Not starting as it isn't finding any useable
>>>> storage pool. - logs at the end of the mail.
>>>>
>>>> I tried deploying a VM using the same storage pool which worked
>>>> wonderfully.
>>>> There i noticed that all my custome storage-offerings are unuseable at
>>>> the moment. They are not accepting any custome storage sizes anymore...
>>>> Throwing failures "VM Creation failed. Volume size: -1GB is out of allowed
>>>> tange. Max:3072 Min:1"...
>>>> So this is also currently broken....
>>>>
>>>> 2022-08-29 10:51:56,189 DEBUG [c.c.v.VmWorkJobDispatcher]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5) Run
>>>> VM work job: com.cloud.vm.VmWorkStart for VM 83, job origin: 3445
>>>> 2022-08-29 10:51:56,191 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>> (logid:ba10fec5) Execute VM work job:
>>>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
>>>> 2022-08-29 10:51:56,192 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>> (logid:ba10fec5) orchestrating VM start for 'v-83-VM'
>>>> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
>>>> 2022-08-29 10:51:56,198 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} state
>>>> transited from [Stopped] to [Starting] with event [StartRequested]. VM's
>>>> original host: null, new host: null, host before state transition: null
>>>> 2022-08-29 10:51:56,198 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>> (logid:ba10fec5) Successfully transitioned to start state for VM instance
>>>> {id: "83", name: "v-83-VM", uuid: "b1d1f9c6-eafe-4527-9335-e649c646aab0",
>>>> type="ConsoleProxy"} reservation id = 6641ccc0-bcef-4b1a-a1f7-8dd8000626ad
>>>> 2022-08-29 10:51:56,203 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>> (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts: null
>>>> 2022-08-29 10:51:56,206 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>> (logid:ba10fec5) VM start attempt #1
>>>> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>> (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
>>>> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>> (logid:ba10fec5) Trying to allocate a host and storage pools from dc:1,
>>>> pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
>>>> 1073741824
>>>> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>> (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
>>>> 2022-08-29 10:51:56,214 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>> (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
>>>> 2022-08-29 10:51:56,215 DEBUG [c.c.d.FirstFitPlanner]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>> (logid:ba10fec5) Searching all possible resources under this Zone: 1
>>>> 2022-08-29 10:51:56,216 DEBUG [c.c.d.FirstFitPlanner]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>> (logid:ba10fec5) Listing clusters in order of aggregate capacity, that have
>>>> (at least one host with) enough CPU and RAM capacity under this Zone: 1
>>>> 2022-08-29 10:51:56,219 DEBUG [c.c.d.FirstFitPlanner]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>> (logid:ba10fec5) Removing from the clusterId list these clusters from avoid
>>>> set: []
>>>> 2022-08-29 10:51:56,224 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>> (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
>>>> 2022-08-29 10:51:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested with
>>>> Custom[UEFI] Boot Type false
>>>> 2022-08-29 10:51:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in dc: 1
>>>> pod:1 cluster:1
>>>> 2022-08-29 10:51:56,227 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4 hosts to
>>>> check for allocation: [Host {"id": "7", "name": "srv-5-comp.blabla.blub",
>>>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
>>>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
>>>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host {"id": "6",
>>>> "name": "srv-3-comp.blabla.blub", "uuid":
>>>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id": "1",
>>>> "name": "srv-2-comp.blabla.blub", "uuid":
>>>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for allocation
>>>> after prioritization: [Host {"id": "7", "name": "srv-5-comp.blabla.blub",
>>>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
>>>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
>>>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host {"id": "6",
>>>> "name": "srv-3-comp.blabla.blub", "uuid":
>>>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id": "1",
>>>> "name": "srv-2-comp.blabla.blub", "uuid":
>>>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for speed=500Mhz,
>>>> Ram=1024 MB
>>>> 2022-08-29 10:51:56,233 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
>>>> srv-5-comp.blabla.blub, uuid: 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is KVM
>>>> hypervisor type, no max guest limit check needed
>>>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu capability
>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
>>>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has enough
>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
>>>> cpuOverprovisioningFactor: 1.0
>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
>>>> and CPU after applying overprovisioning: 127200
>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 , Requested CPU:
>>>> 500
>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
>>>> 189373743104 , Requested RAM: (1.00 GB) 1073741824
>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
>>>> available
>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
>>>> 7, used: 90100, reserved: 0, actual total: 127200, total with
>>>> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
>>>> ,considerReservedCapacity?: true
>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
>>>> 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total: (250.27 GB)
>>>> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
>>>> false , considerReservedCapacity?: true
>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
>>>> list: 7
>>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
>>>> srv-4-comp.blabla.blub, uuid: 134c625d-15ab-461a-a38d-36fb0b979b5d} is KVM
>>>> hypervisor type, no max guest limit check needed
>>>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu capability
>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
>>>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has enough
>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
>>>> cpuOverprovisioningFactor: 1.0
>>>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
>>>> and CPU after applying overprovisioning: 127200
>>>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 , Requested
>>>> CPU: 500
>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
>>>> 251009040384 , Requested RAM: (1.00 GB) 1073741824
>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
>>>> available
>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
>>>> 5, used: 11100, reserved: 0, actual total: 127200, total with
>>>> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
>>>> ,considerReservedCapacity?: true
>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
>>>> 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total: (250.27 GB)
>>>> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
>>>> false , considerReservedCapacity?: true
>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
>>>> list: 5
>>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
>>>> srv-3-comp.blabla.blub, uuid: 6858713b-7e17-46d8-854d-24a13f9c7001} is KVM
>>>> hypervisor type, no max guest limit check needed
>>>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu capability
>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
>>>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has enough
>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
>>>> cpuOverprovisioningFactor: 1.0
>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
>>>> and CPU after applying overprovisioning: 127200
>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 , Requested CPU:
>>>> 500
>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
>>>> 204032835584 , Requested RAM: (1.00 GB) 1073741824
>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
>>>> available
>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
>>>> 6, used: 42900, reserved: 0, actual total: 127200, total with
>>>> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
>>>> ,considerReservedCapacity?: true
>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
>>>> 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total: (250.27 GB)
>>>> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
>>>> false , considerReservedCapacity?: true
>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
>>>> list: 6
>>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
>>>> srv-2-comp.blabla.blub, uuid: 883f7f62-55f2-42f4-8242-16b6847c4204} is KVM
>>>> hypervisor type, no max guest limit check needed
>>>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu capability
>>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
>>>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has enough
>>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
>>>> cpuOverprovisioningFactor: 1.0
>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
>>>> and CPU after applying overprovisioning: 127200
>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 , Requested CPU:
>>>> 500
>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
>>>> 229802639360 , Requested RAM: (1.00 GB) 1073741824
>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
>>>> available
>>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
>>>> 1, used: 48200, reserved: 0, actual total: 127200, total with
>>>> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
>>>> ,considerReservedCapacity?: true
>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
>>>> 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total: (250.27 GB)
>>>> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
>>>> false , considerReservedCapacity?: true
>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
>>>> list: 1
>>>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator returning 4
>>>> suitable hosts
>>>> 2022-08-29 10:51:56,260 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} state
>>>> transited from [Starting] to [Stopped] with event [OperationFailed]. VM's
>>>> original host: null, new host: null, host before state transition: null
>>>> 2022-08-29 10:51:56,260 WARN [c.c.v.VirtualMachineManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>> (logid:ba10fec5) Unable to orchestrate start VM instance {id: "83", name:
>>>> "v-83-VM", uuid: "b1d1f9c6-eafe-4527-9335-e649c646aab0",
>>>> type="ConsoleProxy"} due to [Unable to create deployment, no usable volumes
>>>> found for the VM: 83].
>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to create
>>>> deployment, no usable volumes found for the VM: 83
>>>> at
>>>> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
>>>> at
>>>> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
>>>> at
>>>> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
>>>> at
>>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
>>>> at
>>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
>>>> at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>>>> Source)
>>>> at
>>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>>> at
>>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>>> at
>>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>>> at
>>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>>>> at
>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>>> at
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>>> at
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>>> at
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>>> at
>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>>> at
>>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>>>> at
>>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>>> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>>> at
>>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>> at
>>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>> 2022-08-29 10:51:56,262 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>> (logid:ba10fec5) Done executing VM work job:
>>>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
>>>> 2022-08-29 10:51:56,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>> (logid:ba10fec5) Complete async job-3539, jobStatus: FAILED, resultCode: 0,
>>>> result:
>>>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODMiLCBuYW1lOiAidi04My1WTSIsIHV1aWQ6ICJiMWQxZjljNi1lYWZlLTQ1MjctOTMzNS1lNjQ5YzY0NmFhYjAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODNdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>> (logid:ba10fec5) Publish async job-3539 complete on message bus
>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>> (logid:ba10fec5) Wake up jobs related to job-3539
>>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>> (logid:ba10fec5) Update db status for job-3539
>>>> 2022-08-29 10:51:56,266 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>>> (logid:ba10fec5) Wake up jobs joined with job-3539 and disjoin all subjobs
>>>> created from job- 3539
>>>> 2022-08-29 10:51:56,276 DEBUG [c.c.v.VmWorkJobDispatcher]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5) Done
>>>> with run of VM work job: com.cloud.vm.VmWorkStart for VM 83, job origin:
>>>> 3445
>>>> 2022-08-29 10:51:56,276 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5) Done
>>>> executing com.cloud.vm.VmWorkStart for job-3539
>>>> 2022-08-29 10:51:56,278 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5)
>>>> Remove job-3539 from job monitoring
>>>> 2022-08-29 10:51:56,285 WARN [c.c.c.ConsoleProxyManagerImpl]
>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Unable to allocate console
>>>> proxy standby capacity for zone [1] due to [Unable to orchestrate start VM
>>>> instance {id: "83", name: "v-83-VM", uuid:
>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to [Unable
>>>> to create deployment, no usable volumes found for the VM: 83].].
>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate
>>>> start VM instance {id: "83", name: "v-83-VM", uuid:
>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to [Unable
>>>> to create deployment, no usable volumes found for the VM: 83].
>>>> at
>>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
>>>> at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown
>>>> Source)
>>>> at
>>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>>> at
>>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>>> at
>>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>>> at
>>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>>>> at
>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>>> at
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>>> at
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>>> at
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>>> at
>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>>> at
>>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>>>> at
>>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>>> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>>> at
>>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>> at
>>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>> 2022-08-29 10:51:56,287 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) received console proxy alert
>>>> 2022-08-29 10:51:56,288 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Console proxy creation
>>>> failure, zone: blabla
>>>> 2022-08-29 10:51:56,289 WARN [c.c.a.AlertManagerImpl]
>>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) alertType=[10]
>>>> dataCenterId=[1] podId=[null] clusterId=[null] message=[Console proxy
>>>> creation failure. zone: blabla, error details: Unable to orchestrate start
>>>> VM instance {id: "83", name: "v-83-VM", uuid:
>>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to [Unable
>>>> to create deployment, no usable volumes found for the VM: 83].].
>>>>
>>>>
>>>>
>>>>
>>>> Am Mo., 29. Aug. 2022 um 09:40 Uhr schrieb Slavka Peleva
>>>> <sl...@storpool.com.invalid>:
>>>>
>>>>> Hi Chris,
>>>>>
>>>>> Did you recreate the system VMs? In the 4.17 version, the
>>>>> `systemvm.iso` is
>>>>> deprecated. In its place, you should have the file on agents
>>>>> `/usr/share/cloudstack-common/vms/agent.zip`.
>>>>> Can you share the complete log if the system VMs are newly created
>>>>> after
>>>>> the upgrade?
>>>>>
>>>>> Best regards,
>>>>> Slavka
>>>>>
>>>>> On Sun, Aug 28, 2022 at 2:16 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
>>>>>
>>>>> > Hi everyone,
>>>>> >
>>>>> > faceing some challanges again after upgradeing to 4.17.
>>>>> > Did as explained in the docs.
>>>>> >
>>>>> > Afterwards I am now not able to migrate system VMs to different
>>>>> hosts.
>>>>> > Getting the error
>>>>> > "Exception during migrate: org.libvirt.LibvirtException: Cannot
>>>>> access
>>>>> > storage file '/usr/share/cloudstack-common/vms/systemvm.iso': No
>>>>> such file
>>>>> > or directory"
>>>>> >
>>>>> > Checked the exitstence of the file on the servers:
>>>>> > Mangement-Server has a "systemvm.iso.bak"
>>>>> > Hosts - no "systemvm.iso" at all present.
>>>>> >
>>>>> > Maybe someone has an idea on how to move on with this?
>>>>> >
>>>>> > Regards,
>>>>> > Chris
>>>>> >
>>>>>
>>>>
Re: Error migrate System VMs after upgrageing to 4.17
Posted by Slavka Peleva <sl...@storpool.com.INVALID>.
Thanks, Chris, for the information!
Are you using custom offerings for the system VMs? Can you share more info
if yes?
About the log, it's not the complete log from where the deployment started.
The deployment begins a bit earlier than what you shared. Can you search
the management log for `Sync job-3539 execution on object VmWorkJobQueue`
to get the `logid` for the volume creation (coloured in blue)? And if
possible, share the logs with the found log ID
Here is an example from my logs:
2022-08-29 17:07:49,114 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(consoleproxy-1:ctx-a952a3dc) (logid:89157bb9) Sync job-60484 execution on
object VmWorkJobQueue.4904
2022-08-29 17:07:50,816 DEBUG [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-5:ctx-56172581 job-60479/job-60484 ctx-a900986d)
(logid:95d750ae) Execute VM work job:
com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":4904,"handlerName":"VirtualMachineManagerImpl"}
2022-08-29 17:07:50,817 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-5:ctx-56172581 job-60479/job-60484 ctx-a900986d)
(logid:95d750ae) orchestrating VM start for 'v-4904-VM'
com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
On Mon, Aug 29, 2022 at 2:29 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
> Hi Slavka,
>
> thats what i did after your suggestion / informatin and where the error
> occured.
> Deleted / Destroyed the consoleproxy and waited till recreation. Which
> didn't worked throwing the error i pasted above.
> Sorry for beeing so unspecific.
>
> Regards,
> Chris
>
> Am Mo., 29. Aug. 2022 um 12:46 Uhr schrieb Slavka Peleva <
> slavkap@storpool.com>:
>
>> Hi Chris,
>>
>> I mean to recreate the system VMs by destroying them.
>>
>> On Mon, Aug 29, 2022 at 12:38 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
>>
>>> Hi Slavka,
>>>
>>> didn't tried to restart the SystemVMs. Reading through the docs i had
>>> the impression that "live-patching" would be enough for this....
>>> Anyway - I tried my luck with the console proxy.
>>>
>>> Which failed utterly. Not starting as it isn't finding any useable
>>> storage pool. - logs at the end of the mail.
>>>
>>> I tried deploying a VM using the same storage pool which worked
>>> wonderfully.
>>> There i noticed that all my custome storage-offerings are unuseable at
>>> the moment. They are not accepting any custome storage sizes anymore...
>>> Throwing failures "VM Creation failed. Volume size: -1GB is out of allowed
>>> tange. Max:3072 Min:1"...
>>> So this is also currently broken....
>>>
>>> 2022-08-29 10:51:56,189 DEBUG [c.c.v.VmWorkJobDispatcher]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5) Run
>>> VM work job: com.cloud.vm.VmWorkStart for VM 83, job origin: 3445
>>> 2022-08-29 10:51:56,191 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>> (logid:ba10fec5) Execute VM work job:
>>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
>>> 2022-08-29 10:51:56,192 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>> (logid:ba10fec5) orchestrating VM start for 'v-83-VM'
>>> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
>>> 2022-08-29 10:51:56,198 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} state
>>> transited from [Stopped] to [Starting] with event [StartRequested]. VM's
>>> original host: null, new host: null, host before state transition: null
>>> 2022-08-29 10:51:56,198 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>> (logid:ba10fec5) Successfully transitioned to start state for VM instance
>>> {id: "83", name: "v-83-VM", uuid: "b1d1f9c6-eafe-4527-9335-e649c646aab0",
>>> type="ConsoleProxy"} reservation id = 6641ccc0-bcef-4b1a-a1f7-8dd8000626ad
>>> 2022-08-29 10:51:56,203 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>> (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts: null
>>> 2022-08-29 10:51:56,206 DEBUG [c.c.v.VirtualMachineManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>> (logid:ba10fec5) VM start attempt #1
>>> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>> (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
>>> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>> (logid:ba10fec5) Trying to allocate a host and storage pools from dc:1,
>>> pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
>>> 1073741824
>>> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>> (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
>>> 2022-08-29 10:51:56,214 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>> (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
>>> 2022-08-29 10:51:56,215 DEBUG [c.c.d.FirstFitPlanner]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>> (logid:ba10fec5) Searching all possible resources under this Zone: 1
>>> 2022-08-29 10:51:56,216 DEBUG [c.c.d.FirstFitPlanner]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>> (logid:ba10fec5) Listing clusters in order of aggregate capacity, that have
>>> (at least one host with) enough CPU and RAM capacity under this Zone: 1
>>> 2022-08-29 10:51:56,219 DEBUG [c.c.d.FirstFitPlanner]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>> (logid:ba10fec5) Removing from the clusterId list these clusters from avoid
>>> set: []
>>> 2022-08-29 10:51:56,224 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>> (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
>>> 2022-08-29 10:51:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested with
>>> Custom[UEFI] Boot Type false
>>> 2022-08-29 10:51:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in dc: 1
>>> pod:1 cluster:1
>>> 2022-08-29 10:51:56,227 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4 hosts to
>>> check for allocation: [Host {"id": "7", "name": "srv-5-comp.blabla.blub",
>>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
>>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
>>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host {"id": "6",
>>> "name": "srv-3-comp.blabla.blub", "uuid":
>>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id": "1",
>>> "name": "srv-2-comp.blabla.blub", "uuid":
>>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
>>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for allocation
>>> after prioritization: [Host {"id": "7", "name": "srv-5-comp.blabla.blub",
>>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
>>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
>>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host {"id": "6",
>>> "name": "srv-3-comp.blabla.blub", "uuid":
>>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id": "1",
>>> "name": "srv-2-comp.blabla.blub", "uuid":
>>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
>>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for speed=500Mhz,
>>> Ram=1024 MB
>>> 2022-08-29 10:51:56,233 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
>>> srv-5-comp.blabla.blub, uuid: 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is KVM
>>> hypervisor type, no max guest limit check needed
>>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu capability
>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
>>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has enough
>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
>>> cpuOverprovisioningFactor: 1.0
>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
>>> and CPU after applying overprovisioning: 127200
>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 , Requested CPU:
>>> 500
>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
>>> 189373743104 , Requested RAM: (1.00 GB) 1073741824
>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
>>> available
>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
>>> 7, used: 90100, reserved: 0, actual total: 127200, total with
>>> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
>>> ,considerReservedCapacity?: true
>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
>>> 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total: (250.27 GB)
>>> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
>>> false , considerReservedCapacity?: true
>>> 2022-08-29 10:51:56,237 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
>>> list: 7
>>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
>>> srv-4-comp.blabla.blub, uuid: 134c625d-15ab-461a-a38d-36fb0b979b5d} is KVM
>>> hypervisor type, no max guest limit check needed
>>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu capability
>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
>>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has enough
>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
>>> cpuOverprovisioningFactor: 1.0
>>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
>>> and CPU after applying overprovisioning: 127200
>>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 , Requested
>>> CPU: 500
>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
>>> 251009040384 , Requested RAM: (1.00 GB) 1073741824
>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
>>> available
>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
>>> 5, used: 11100, reserved: 0, actual total: 127200, total with
>>> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
>>> ,considerReservedCapacity?: true
>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
>>> 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total: (250.27 GB)
>>> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
>>> false , considerReservedCapacity?: true
>>> 2022-08-29 10:51:56,241 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
>>> list: 5
>>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
>>> srv-3-comp.blabla.blub, uuid: 6858713b-7e17-46d8-854d-24a13f9c7001} is KVM
>>> hypervisor type, no max guest limit check needed
>>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu capability
>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
>>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has enough
>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
>>> cpuOverprovisioningFactor: 1.0
>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
>>> and CPU after applying overprovisioning: 127200
>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 , Requested CPU:
>>> 500
>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
>>> 204032835584 , Requested RAM: (1.00 GB) 1073741824
>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
>>> available
>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
>>> 6, used: 42900, reserved: 0, actual total: 127200, total with
>>> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
>>> ,considerReservedCapacity?: true
>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
>>> 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total: (250.27 GB)
>>> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
>>> false , considerReservedCapacity?: true
>>> 2022-08-29 10:51:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
>>> list: 6
>>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
>>> srv-2-comp.blabla.blub, uuid: 883f7f62-55f2-42f4-8242-16b6847c4204} is KVM
>>> hypervisor type, no max guest limit check needed
>>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu capability
>>> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
>>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has enough
>>> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
>>> cpuOverprovisioningFactor: 1.0
>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
>>> and CPU after applying overprovisioning: 127200
>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 , Requested CPU:
>>> 500
>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
>>> 229802639360 , Requested RAM: (1.00 GB) 1073741824
>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
>>> available
>>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
>>> 1, used: 48200, reserved: 0, actual total: 127200, total with
>>> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
>>> ,considerReservedCapacity?: true
>>> 2022-08-29 10:51:56,249 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
>>> 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total: (250.27 GB)
>>> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
>>> false , considerReservedCapacity?: true
>>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
>>> list: 1
>>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>>> FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator returning 4
>>> suitable hosts
>>> 2022-08-29 10:51:56,260 DEBUG [c.c.c.CapacityManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} state
>>> transited from [Starting] to [Stopped] with event [OperationFailed]. VM's
>>> original host: null, new host: null, host before state transition: null
>>> 2022-08-29 10:51:56,260 WARN [c.c.v.VirtualMachineManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>> (logid:ba10fec5) Unable to orchestrate start VM instance {id: "83", name:
>>> "v-83-VM", uuid: "b1d1f9c6-eafe-4527-9335-e649c646aab0",
>>> type="ConsoleProxy"} due to [Unable to create deployment, no usable volumes
>>> found for the VM: 83].
>>> com.cloud.utils.exception.CloudRuntimeException: Unable to create
>>> deployment, no usable volumes found for the VM: 83
>>> at
>>> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
>>> at
>>> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
>>> at
>>> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
>>> at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
>>> at
>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>> at
>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>> at
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>> at
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>>> at
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>> 2022-08-29 10:51:56,262 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>> (logid:ba10fec5) Done executing VM work job:
>>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
>>> 2022-08-29 10:51:56,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>> (logid:ba10fec5) Complete async job-3539, jobStatus: FAILED, resultCode: 0,
>>> result:
>>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODMiLCBuYW1lOiAidi04My1WTSIsIHV1aWQ6ICJiMWQxZjljNi1lYWZlLTQ1MjctOTMzNS1lNjQ5YzY0NmFhYjAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODNdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>> (logid:ba10fec5) Publish async job-3539 complete on message bus
>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>> (logid:ba10fec5) Wake up jobs related to job-3539
>>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>> (logid:ba10fec5) Update db status for job-3539
>>> 2022-08-29 10:51:56,266 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>>> (logid:ba10fec5) Wake up jobs joined with job-3539 and disjoin all subjobs
>>> created from job- 3539
>>> 2022-08-29 10:51:56,276 DEBUG [c.c.v.VmWorkJobDispatcher]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5) Done
>>> with run of VM work job: com.cloud.vm.VmWorkStart for VM 83, job origin:
>>> 3445
>>> 2022-08-29 10:51:56,276 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5) Done
>>> executing com.cloud.vm.VmWorkStart for job-3539
>>> 2022-08-29 10:51:56,278 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5)
>>> Remove job-3539 from job monitoring
>>> 2022-08-29 10:51:56,285 WARN [c.c.c.ConsoleProxyManagerImpl]
>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Unable to allocate console
>>> proxy standby capacity for zone [1] due to [Unable to orchestrate start VM
>>> instance {id: "83", name: "v-83-VM", uuid:
>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to [Unable
>>> to create deployment, no usable volumes found for the VM: 83].].
>>> com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate
>>> start VM instance {id: "83", name: "v-83-VM", uuid:
>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to [Unable
>>> to create deployment, no usable volumes found for the VM: 83].
>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
>>> at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
>>> at
>>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>>> at
>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>> at
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>>> at
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>>> at
>>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>>> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>> 2022-08-29 10:51:56,287 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) received console proxy alert
>>> 2022-08-29 10:51:56,288 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Console proxy creation
>>> failure, zone: blabla
>>> 2022-08-29 10:51:56,289 WARN [c.c.a.AlertManagerImpl]
>>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) alertType=[10]
>>> dataCenterId=[1] podId=[null] clusterId=[null] message=[Console proxy
>>> creation failure. zone: blabla, error details: Unable to orchestrate start
>>> VM instance {id: "83", name: "v-83-VM", uuid:
>>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to [Unable
>>> to create deployment, no usable volumes found for the VM: 83].].
>>>
>>>
>>>
>>>
>>> Am Mo., 29. Aug. 2022 um 09:40 Uhr schrieb Slavka Peleva
>>> <sl...@storpool.com.invalid>:
>>>
>>>> Hi Chris,
>>>>
>>>> Did you recreate the system VMs? In the 4.17 version, the
>>>> `systemvm.iso` is
>>>> deprecated. In its place, you should have the file on agents
>>>> `/usr/share/cloudstack-common/vms/agent.zip`.
>>>> Can you share the complete log if the system VMs are newly created after
>>>> the upgrade?
>>>>
>>>> Best regards,
>>>> Slavka
>>>>
>>>> On Sun, Aug 28, 2022 at 2:16 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
>>>>
>>>> > Hi everyone,
>>>> >
>>>> > faceing some challanges again after upgradeing to 4.17.
>>>> > Did as explained in the docs.
>>>> >
>>>> > Afterwards I am now not able to migrate system VMs to different hosts.
>>>> > Getting the error
>>>> > "Exception during migrate: org.libvirt.LibvirtException: Cannot access
>>>> > storage file '/usr/share/cloudstack-common/vms/systemvm.iso': No such
>>>> file
>>>> > or directory"
>>>> >
>>>> > Checked the exitstence of the file on the servers:
>>>> > Mangement-Server has a "systemvm.iso.bak"
>>>> > Hosts - no "systemvm.iso" at all present.
>>>> >
>>>> > Maybe someone has an idea on how to move on with this?
>>>> >
>>>> > Regards,
>>>> > Chris
>>>> >
>>>>
>>>
Re: Error migrate System VMs after upgrageing to 4.17
Posted by "Vash_X@gmx.de" <Va...@gmx.de>.
Hi Slavka,
thats what i did after your suggestion / informatin and where the error
occured.
Deleted / Destroyed the consoleproxy and waited till recreation. Which
didn't worked throwing the error i pasted above.
Sorry for beeing so unspecific.
Regards,
Chris
Am Mo., 29. Aug. 2022 um 12:46 Uhr schrieb Slavka Peleva <
slavkap@storpool.com>:
> Hi Chris,
>
> I mean to recreate the system VMs by destroying them.
>
> On Mon, Aug 29, 2022 at 12:38 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
>
>> Hi Slavka,
>>
>> didn't tried to restart the SystemVMs. Reading through the docs i had the
>> impression that "live-patching" would be enough for this....
>> Anyway - I tried my luck with the console proxy.
>>
>> Which failed utterly. Not starting as it isn't finding any useable
>> storage pool. - logs at the end of the mail.
>>
>> I tried deploying a VM using the same storage pool which worked
>> wonderfully.
>> There i noticed that all my custome storage-offerings are unuseable at
>> the moment. They are not accepting any custome storage sizes anymore...
>> Throwing failures "VM Creation failed. Volume size: -1GB is out of allowed
>> tange. Max:3072 Min:1"...
>> So this is also currently broken....
>>
>> 2022-08-29 10:51:56,189 DEBUG [c.c.v.VmWorkJobDispatcher]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5) Run
>> VM work job: com.cloud.vm.VmWorkStart for VM 83, job origin: 3445
>> 2022-08-29 10:51:56,191 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>> (logid:ba10fec5) Execute VM work job:
>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
>> 2022-08-29 10:51:56,192 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>> (logid:ba10fec5) orchestrating VM start for 'v-83-VM'
>> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
>> 2022-08-29 10:51:56,198 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} state
>> transited from [Stopped] to [Starting] with event [StartRequested]. VM's
>> original host: null, new host: null, host before state transition: null
>> 2022-08-29 10:51:56,198 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>> (logid:ba10fec5) Successfully transitioned to start state for VM instance
>> {id: "83", name: "v-83-VM", uuid: "b1d1f9c6-eafe-4527-9335-e649c646aab0",
>> type="ConsoleProxy"} reservation id = 6641ccc0-bcef-4b1a-a1f7-8dd8000626ad
>> 2022-08-29 10:51:56,203 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>> (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts: null
>> 2022-08-29 10:51:56,206 DEBUG [c.c.v.VirtualMachineManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>> (logid:ba10fec5) VM start attempt #1
>> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>> (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
>> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>> (logid:ba10fec5) Trying to allocate a host and storage pools from dc:1,
>> pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
>> 1073741824
>> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>> (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
>> 2022-08-29 10:51:56,214 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>> (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
>> 2022-08-29 10:51:56,215 DEBUG [c.c.d.FirstFitPlanner]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>> (logid:ba10fec5) Searching all possible resources under this Zone: 1
>> 2022-08-29 10:51:56,216 DEBUG [c.c.d.FirstFitPlanner]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>> (logid:ba10fec5) Listing clusters in order of aggregate capacity, that have
>> (at least one host with) enough CPU and RAM capacity under this Zone: 1
>> 2022-08-29 10:51:56,219 DEBUG [c.c.d.FirstFitPlanner]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>> (logid:ba10fec5) Removing from the clusterId list these clusters from avoid
>> set: []
>> 2022-08-29 10:51:56,224 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>> (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
>> 2022-08-29 10:51:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested with
>> Custom[UEFI] Boot Type false
>> 2022-08-29 10:51:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in dc: 1
>> pod:1 cluster:1
>> 2022-08-29 10:51:56,227 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4 hosts to
>> check for allocation: [Host {"id": "7", "name": "srv-5-comp.blabla.blub",
>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host {"id": "6",
>> "name": "srv-3-comp.blabla.blub", "uuid":
>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id": "1",
>> "name": "srv-2-comp.blabla.blub", "uuid":
>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for allocation
>> after prioritization: [Host {"id": "7", "name": "srv-5-comp.blabla.blub",
>> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
>> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
>> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host {"id": "6",
>> "name": "srv-3-comp.blabla.blub", "uuid":
>> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id": "1",
>> "name": "srv-2-comp.blabla.blub", "uuid":
>> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
>> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for speed=500Mhz,
>> Ram=1024 MB
>> 2022-08-29 10:51:56,233 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
>> srv-5-comp.blabla.blub, uuid: 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is KVM
>> hypervisor type, no max guest limit check needed
>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu capability
>> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
>> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has enough
>> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
>> cpuOverprovisioningFactor: 1.0
>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
>> and CPU after applying overprovisioning: 127200
>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 , Requested CPU:
>> 500
>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
>> 189373743104 , Requested RAM: (1.00 GB) 1073741824
>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
>> available
>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
>> 7, used: 90100, reserved: 0, actual total: 127200, total with
>> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
>> ,considerReservedCapacity?: true
>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
>> 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total: (250.27 GB)
>> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
>> false , considerReservedCapacity?: true
>> 2022-08-29 10:51:56,237 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
>> list: 7
>> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
>> srv-4-comp.blabla.blub, uuid: 134c625d-15ab-461a-a38d-36fb0b979b5d} is KVM
>> hypervisor type, no max guest limit check needed
>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu capability
>> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
>> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has enough
>> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
>> cpuOverprovisioningFactor: 1.0
>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
>> and CPU after applying overprovisioning: 127200
>> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 , Requested
>> CPU: 500
>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
>> 251009040384 , Requested RAM: (1.00 GB) 1073741824
>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
>> available
>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
>> 5, used: 11100, reserved: 0, actual total: 127200, total with
>> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
>> ,considerReservedCapacity?: true
>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
>> 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total: (250.27 GB)
>> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
>> false , considerReservedCapacity?: true
>> 2022-08-29 10:51:56,241 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
>> list: 5
>> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
>> srv-3-comp.blabla.blub, uuid: 6858713b-7e17-46d8-854d-24a13f9c7001} is KVM
>> hypervisor type, no max guest limit check needed
>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu capability
>> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
>> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has enough
>> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
>> cpuOverprovisioningFactor: 1.0
>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
>> and CPU after applying overprovisioning: 127200
>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 , Requested CPU:
>> 500
>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
>> 204032835584 , Requested RAM: (1.00 GB) 1073741824
>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
>> available
>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
>> 6, used: 42900, reserved: 0, actual total: 127200, total with
>> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
>> ,considerReservedCapacity?: true
>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
>> 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total: (250.27 GB)
>> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
>> false , considerReservedCapacity?: true
>> 2022-08-29 10:51:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
>> list: 6
>> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
>> srv-2-comp.blabla.blub, uuid: 883f7f62-55f2-42f4-8242-16b6847c4204} is KVM
>> hypervisor type, no max guest limit check needed
>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu capability
>> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
>> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has enough
>> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
>> cpuOverprovisioningFactor: 1.0
>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
>> and CPU after applying overprovisioning: 127200
>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 , Requested CPU:
>> 500
>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
>> 229802639360 , Requested RAM: (1.00 GB) 1073741824
>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
>> available
>> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
>> 1, used: 48200, reserved: 0, actual total: 127200, total with
>> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
>> ,considerReservedCapacity?: true
>> 2022-08-29 10:51:56,249 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
>> 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total: (250.27 GB)
>> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
>> false , considerReservedCapacity?: true
>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
>> list: 1
>> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
>> FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator returning 4
>> suitable hosts
>> 2022-08-29 10:51:56,260 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} state
>> transited from [Starting] to [Stopped] with event [OperationFailed]. VM's
>> original host: null, new host: null, host before state transition: null
>> 2022-08-29 10:51:56,260 WARN [c.c.v.VirtualMachineManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>> (logid:ba10fec5) Unable to orchestrate start VM instance {id: "83", name:
>> "v-83-VM", uuid: "b1d1f9c6-eafe-4527-9335-e649c646aab0",
>> type="ConsoleProxy"} due to [Unable to create deployment, no usable volumes
>> found for the VM: 83].
>> com.cloud.utils.exception.CloudRuntimeException: Unable to create
>> deployment, no usable volumes found for the VM: 83
>> at
>> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
>> at
>> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
>> at
>> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
>> at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
>> at
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> at
>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>> at
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> at java.base/java.lang.Thread.run(Thread.java:829)
>> 2022-08-29 10:51:56,262 DEBUG [c.c.v.VmWorkJobHandlerProxy]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>> (logid:ba10fec5) Done executing VM work job:
>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
>> 2022-08-29 10:51:56,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>> (logid:ba10fec5) Complete async job-3539, jobStatus: FAILED, resultCode: 0,
>> result:
>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODMiLCBuYW1lOiAidi04My1WTSIsIHV1aWQ6ICJiMWQxZjljNi1lYWZlLTQ1MjctOTMzNS1lNjQ5YzY0NmFhYjAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODNdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>> (logid:ba10fec5) Publish async job-3539 complete on message bus
>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>> (logid:ba10fec5) Wake up jobs related to job-3539
>> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>> (logid:ba10fec5) Update db status for job-3539
>> 2022-08-29 10:51:56,266 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
>> (logid:ba10fec5) Wake up jobs joined with job-3539 and disjoin all subjobs
>> created from job- 3539
>> 2022-08-29 10:51:56,276 DEBUG [c.c.v.VmWorkJobDispatcher]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5) Done
>> with run of VM work job: com.cloud.vm.VmWorkStart for VM 83, job origin:
>> 3445
>> 2022-08-29 10:51:56,276 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5) Done
>> executing com.cloud.vm.VmWorkStart for job-3539
>> 2022-08-29 10:51:56,278 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5)
>> Remove job-3539 from job monitoring
>> 2022-08-29 10:51:56,285 WARN [c.c.c.ConsoleProxyManagerImpl]
>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Unable to allocate console
>> proxy standby capacity for zone [1] due to [Unable to orchestrate start VM
>> instance {id: "83", name: "v-83-VM", uuid:
>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to [Unable
>> to create deployment, no usable volumes found for the VM: 83].].
>> com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate
>> start VM instance {id: "83", name: "v-83-VM", uuid:
>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to [Unable
>> to create deployment, no usable volumes found for the VM: 83].
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
>> at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
>> at
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>> at
>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>> at
>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
>> at
>> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>> at
>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>> at java.base/java.lang.Thread.run(Thread.java:829)
>> 2022-08-29 10:51:56,287 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) received console proxy alert
>> 2022-08-29 10:51:56,288 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Console proxy creation
>> failure, zone: blabla
>> 2022-08-29 10:51:56,289 WARN [c.c.a.AlertManagerImpl]
>> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) alertType=[10]
>> dataCenterId=[1] podId=[null] clusterId=[null] message=[Console proxy
>> creation failure. zone: blabla, error details: Unable to orchestrate start
>> VM instance {id: "83", name: "v-83-VM", uuid:
>> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to [Unable
>> to create deployment, no usable volumes found for the VM: 83].].
>>
>>
>>
>>
>> Am Mo., 29. Aug. 2022 um 09:40 Uhr schrieb Slavka Peleva
>> <sl...@storpool.com.invalid>:
>>
>>> Hi Chris,
>>>
>>> Did you recreate the system VMs? In the 4.17 version, the `systemvm.iso`
>>> is
>>> deprecated. In its place, you should have the file on agents
>>> `/usr/share/cloudstack-common/vms/agent.zip`.
>>> Can you share the complete log if the system VMs are newly created after
>>> the upgrade?
>>>
>>> Best regards,
>>> Slavka
>>>
>>> On Sun, Aug 28, 2022 at 2:16 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
>>>
>>> > Hi everyone,
>>> >
>>> > faceing some challanges again after upgradeing to 4.17.
>>> > Did as explained in the docs.
>>> >
>>> > Afterwards I am now not able to migrate system VMs to different hosts.
>>> > Getting the error
>>> > "Exception during migrate: org.libvirt.LibvirtException: Cannot access
>>> > storage file '/usr/share/cloudstack-common/vms/systemvm.iso': No such
>>> file
>>> > or directory"
>>> >
>>> > Checked the exitstence of the file on the servers:
>>> > Mangement-Server has a "systemvm.iso.bak"
>>> > Hosts - no "systemvm.iso" at all present.
>>> >
>>> > Maybe someone has an idea on how to move on with this?
>>> >
>>> > Regards,
>>> > Chris
>>> >
>>>
>>
Re: Error migrate System VMs after upgrageing to 4.17
Posted by Slavka Peleva <sl...@storpool.com.INVALID>.
Hi Chris,
I mean to recreate the system VMs by destroying them.
On Mon, Aug 29, 2022 at 12:38 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
> Hi Slavka,
>
> didn't tried to restart the SystemVMs. Reading through the docs i had the
> impression that "live-patching" would be enough for this....
> Anyway - I tried my luck with the console proxy.
>
> Which failed utterly. Not starting as it isn't finding any useable storage
> pool. - logs at the end of the mail.
>
> I tried deploying a VM using the same storage pool which worked
> wonderfully.
> There i noticed that all my custome storage-offerings are unuseable at the
> moment. They are not accepting any custome storage sizes anymore...
> Throwing failures "VM Creation failed. Volume size: -1GB is out of allowed
> tange. Max:3072 Min:1"...
> So this is also currently broken....
>
> 2022-08-29 10:51:56,189 DEBUG [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5) Run
> VM work job: com.cloud.vm.VmWorkStart for VM 83, job origin: 3445
> 2022-08-29 10:51:56,191 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> (logid:ba10fec5) Execute VM work job:
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
> 2022-08-29 10:51:56,192 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> (logid:ba10fec5) orchestrating VM start for 'v-83-VM'
> com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
> 2022-08-29 10:51:56,198 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} state
> transited from [Stopped] to [Starting] with event [StartRequested]. VM's
> original host: null, new host: null, host before state transition: null
> 2022-08-29 10:51:56,198 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> (logid:ba10fec5) Successfully transitioned to start state for VM instance
> {id: "83", name: "v-83-VM", uuid: "b1d1f9c6-eafe-4527-9335-e649c646aab0",
> type="ConsoleProxy"} reservation id = 6641ccc0-bcef-4b1a-a1f7-8dd8000626ad
> 2022-08-29 10:51:56,203 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> (logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts: null
> 2022-08-29 10:51:56,206 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> (logid:ba10fec5) VM start attempt #1
> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> (logid:ba10fec5) DeploymentPlanner allocation algorithm: null
> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> (logid:ba10fec5) Trying to allocate a host and storage pools from dc:1,
> pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
> 1073741824
> 2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> (logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
> 2022-08-29 10:51:56,214 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> (logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
> 2022-08-29 10:51:56,215 DEBUG [c.c.d.FirstFitPlanner]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> (logid:ba10fec5) Searching all possible resources under this Zone: 1
> 2022-08-29 10:51:56,216 DEBUG [c.c.d.FirstFitPlanner]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> (logid:ba10fec5) Listing clusters in order of aggregate capacity, that have
> (at least one host with) enough CPU and RAM capacity under this Zone: 1
> 2022-08-29 10:51:56,219 DEBUG [c.c.d.FirstFitPlanner]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> (logid:ba10fec5) Removing from the clusterId list these clusters from avoid
> set: []
> 2022-08-29 10:51:56,224 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> (logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
> 2022-08-29 10:51:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested with
> Custom[UEFI] Boot Type false
> 2022-08-29 10:51:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in dc: 1
> pod:1 cluster:1
> 2022-08-29 10:51:56,227 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4 hosts to
> check for allocation: [Host {"id": "7", "name": "srv-5-comp.blabla.blub",
> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host {"id": "6",
> "name": "srv-3-comp.blabla.blub", "uuid":
> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id": "1",
> "name": "srv-2-comp.blabla.blub", "uuid":
> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for allocation
> after prioritization: [Host {"id": "7", "name": "srv-5-comp.blabla.blub",
> "uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
> {"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
> "134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host {"id": "6",
> "name": "srv-3-comp.blabla.blub", "uuid":
> "6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id": "1",
> "name": "srv-2-comp.blabla.blub", "uuid":
> "883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
> 2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Looking for speed=500Mhz,
> Ram=1024 MB
> 2022-08-29 10:51:56,233 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
> srv-5-comp.blabla.blub, uuid: 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is KVM
> hypervisor type, no max guest limit check needed
> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu capability
> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> 2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has enough
> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> cpuOverprovisioningFactor: 1.0
> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
> and CPU after applying overprovisioning: 127200
> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 , Requested CPU:
> 500
> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
> 189373743104 , Requested RAM: (1.00 GB) 1073741824
> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> available
> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
> 7, used: 90100, reserved: 0, actual total: 127200, total with
> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> ,considerReservedCapacity?: true
> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
> 7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total: (250.27 GB)
> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> false , considerReservedCapacity?: true
> 2022-08-29 10:51:56,237 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
> list: 7
> 2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
> srv-4-comp.blabla.blub, uuid: 134c625d-15ab-461a-a38d-36fb0b979b5d} is KVM
> hypervisor type, no max guest limit check needed
> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu capability
> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> 2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has enough
> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> cpuOverprovisioningFactor: 1.0
> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
> and CPU after applying overprovisioning: 127200
> 2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 , Requested
> CPU: 500
> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
> 251009040384 , Requested RAM: (1.00 GB) 1073741824
> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> available
> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
> 5, used: 11100, reserved: 0, actual total: 127200, total with
> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> ,considerReservedCapacity?: true
> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
> 5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total: (250.27 GB)
> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> false , considerReservedCapacity?: true
> 2022-08-29 10:51:56,241 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
> list: 5
> 2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
> srv-3-comp.blabla.blub, uuid: 6858713b-7e17-46d8-854d-24a13f9c7001} is KVM
> hypervisor type, no max guest limit check needed
> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu capability
> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> 2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has enough
> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> cpuOverprovisioningFactor: 1.0
> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
> and CPU after applying overprovisioning: 127200
> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 , Requested CPU:
> 500
> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
> 204032835584 , Requested RAM: (1.00 GB) 1073741824
> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> available
> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
> 6, used: 42900, reserved: 0, actual total: 127200, total with
> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> ,considerReservedCapacity?: true
> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
> 6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total: (250.27 GB)
> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> false , considerReservedCapacity?: true
> 2022-08-29 10:51:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
> list: 6
> 2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
> srv-2-comp.blabla.blub, uuid: 883f7f62-55f2-42f4-8242-16b6847c4204} is KVM
> hypervisor type, no max guest limit check needed
> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu capability
> (cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
> 2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has enough
> capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
> cpuOverprovisioningFactor: 1.0
> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
> and CPU after applying overprovisioning: 127200
> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 , Requested CPU:
> 500
> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
> 229802639360 , Requested RAM: (1.00 GB) 1073741824
> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
> available
> 2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
> 1, used: 48200, reserved: 0, actual total: 127200, total with
> overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
> ,considerReservedCapacity?: true
> 2022-08-29 10:51:56,249 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
> 1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total: (250.27 GB)
> 268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
> false , considerReservedCapacity?: true
> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
> list: 1
> 2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
> FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator returning 4
> suitable hosts
> 2022-08-29 10:51:56,260 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> (logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} state
> transited from [Starting] to [Stopped] with event [OperationFailed]. VM's
> original host: null, new host: null, host before state transition: null
> 2022-08-29 10:51:56,260 WARN [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> (logid:ba10fec5) Unable to orchestrate start VM instance {id: "83", name:
> "v-83-VM", uuid: "b1d1f9c6-eafe-4527-9335-e649c646aab0",
> type="ConsoleProxy"} due to [Unable to create deployment, no usable volumes
> found for the VM: 83].
> com.cloud.utils.exception.CloudRuntimeException: Unable to create
> deployment, no usable volumes found for the VM: 83
> at
> com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
> at
> com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
> at
> com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
> at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> 2022-08-29 10:51:56,262 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> (logid:ba10fec5) Done executing VM work job:
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
> 2022-08-29 10:51:56,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> (logid:ba10fec5) Complete async job-3539, jobStatus: FAILED, resultCode: 0,
> result:
> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODMiLCBuYW1lOiAidi04My1WTSIsIHV1aWQ6ICJiMWQxZjljNi1lYWZlLTQ1MjctOTMzNS1lNjQ5YzY0NmFhYjAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODNdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> (logid:ba10fec5) Publish async job-3539 complete on message bus
> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> (logid:ba10fec5) Wake up jobs related to job-3539
> 2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> (logid:ba10fec5) Update db status for job-3539
> 2022-08-29 10:51:56,266 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
> (logid:ba10fec5) Wake up jobs joined with job-3539 and disjoin all subjobs
> created from job- 3539
> 2022-08-29 10:51:56,276 DEBUG [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5) Done
> with run of VM work job: com.cloud.vm.VmWorkStart for VM 83, job origin:
> 3445
> 2022-08-29 10:51:56,276 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5) Done
> executing com.cloud.vm.VmWorkStart for job-3539
> 2022-08-29 10:51:56,278 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5)
> Remove job-3539 from job monitoring
> 2022-08-29 10:51:56,285 WARN [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Unable to allocate console
> proxy standby capacity for zone [1] due to [Unable to orchestrate start VM
> instance {id: "83", name: "v-83-VM", uuid:
> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to [Unable
> to create deployment, no usable volumes found for the VM: 83].].
> com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate
> start VM instance {id: "83", name: "v-83-VM", uuid:
> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to [Unable
> to create deployment, no usable volumes found for the VM: 83].
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
> at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> 2022-08-29 10:51:56,287 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) received console proxy alert
> 2022-08-29 10:51:56,288 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Console proxy creation
> failure, zone: blabla
> 2022-08-29 10:51:56,289 WARN [c.c.a.AlertManagerImpl]
> (consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) alertType=[10]
> dataCenterId=[1] podId=[null] clusterId=[null] message=[Console proxy
> creation failure. zone: blabla, error details: Unable to orchestrate start
> VM instance {id: "83", name: "v-83-VM", uuid:
> "b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to [Unable
> to create deployment, no usable volumes found for the VM: 83].].
>
>
>
>
> Am Mo., 29. Aug. 2022 um 09:40 Uhr schrieb Slavka Peleva
> <sl...@storpool.com.invalid>:
>
>> Hi Chris,
>>
>> Did you recreate the system VMs? In the 4.17 version, the `systemvm.iso`
>> is
>> deprecated. In its place, you should have the file on agents
>> `/usr/share/cloudstack-common/vms/agent.zip`.
>> Can you share the complete log if the system VMs are newly created after
>> the upgrade?
>>
>> Best regards,
>> Slavka
>>
>> On Sun, Aug 28, 2022 at 2:16 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
>>
>> > Hi everyone,
>> >
>> > faceing some challanges again after upgradeing to 4.17.
>> > Did as explained in the docs.
>> >
>> > Afterwards I am now not able to migrate system VMs to different hosts.
>> > Getting the error
>> > "Exception during migrate: org.libvirt.LibvirtException: Cannot access
>> > storage file '/usr/share/cloudstack-common/vms/systemvm.iso': No such
>> file
>> > or directory"
>> >
>> > Checked the exitstence of the file on the servers:
>> > Mangement-Server has a "systemvm.iso.bak"
>> > Hosts - no "systemvm.iso" at all present.
>> >
>> > Maybe someone has an idea on how to move on with this?
>> >
>> > Regards,
>> > Chris
>> >
>>
>
Re: Error migrate System VMs after upgrageing to 4.17
Posted by "Vash_X@gmx.de" <Va...@gmx.de>.
Hi Slavka,
didn't tried to restart the SystemVMs. Reading through the docs i had the
impression that "live-patching" would be enough for this....
Anyway - I tried my luck with the console proxy.
Which failed utterly. Not starting as it isn't finding any useable storage
pool. - logs at the end of the mail.
I tried deploying a VM using the same storage pool which worked wonderfully.
There i noticed that all my custome storage-offerings are unuseable at the
moment. They are not accepting any custome storage sizes anymore...
Throwing failures "VM Creation failed. Volume size: -1GB is out of allowed
tange. Max:3072 Min:1"...
So this is also currently broken....
2022-08-29 10:51:56,189 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5) Run
VM work job: com.cloud.vm.VmWorkStart for VM 83, job origin: 3445
2022-08-29 10:51:56,191 DEBUG [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
(logid:ba10fec5) Execute VM work job:
com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
2022-08-29 10:51:56,192 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
(logid:ba10fec5) orchestrating VM start for 'v-83-VM'
com.cloud.vm.VirtualMachineProfile$Param@b66cdd7d set to null
2022-08-29 10:51:56,198 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
(logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
"b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} state
transited from [Stopped] to [Starting] with event [StartRequested]. VM's
original host: null, new host: null, host before state transition: null
2022-08-29 10:51:56,198 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
(logid:ba10fec5) Successfully transitioned to start state for VM instance
{id: "83", name: "v-83-VM", uuid: "b1d1f9c6-eafe-4527-9335-e649c646aab0",
type="ConsoleProxy"} reservation id = 6641ccc0-bcef-4b1a-a1f7-8dd8000626ad
2022-08-29 10:51:56,203 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
(logid:ba10fec5) Deploy avoids pods: null, clusters: null, hosts: null
2022-08-29 10:51:56,206 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
(logid:ba10fec5) VM start attempt #1
2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
(logid:ba10fec5) DeploymentPlanner allocation algorithm: null
2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
(logid:ba10fec5) Trying to allocate a host and storage pools from dc:1,
pod:null,cluster:null, requested cpu: 500, requested ram: (1.00 GB)
1073741824
2022-08-29 10:51:56,207 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
(logid:ba10fec5) Is ROOT volume READY (pool already allocated)?: No
2022-08-29 10:51:56,214 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
(logid:ba10fec5) Deploy avoids pods: [], clusters: [], hosts: null
2022-08-29 10:51:56,215 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
(logid:ba10fec5) Searching all possible resources under this Zone: 1
2022-08-29 10:51:56,216 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
(logid:ba10fec5) Listing clusters in order of aggregate capacity, that have
(at least one host with) enough CPU and RAM capacity under this Zone: 1
2022-08-29 10:51:56,219 DEBUG [c.c.d.FirstFitPlanner]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
(logid:ba10fec5) Removing from the clusterId list these clusters from avoid
set: []
2022-08-29 10:51:56,224 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
(logid:ba10fec5) Checking resources in Cluster: 1 under Pod: 1
2022-08-29 10:51:56,225 INFO [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Guest VM is requested with
Custom[UEFI] Boot Type false
2022-08-29 10:51:56,225 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Looking for hosts in dc: 1
pod:1 cluster:1
2022-08-29 10:51:56,227 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) FirstFitAllocator has 4 hosts to
check for allocation: [Host {"id": "7", "name": "srv-5-comp.blabla.blub",
"uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
{"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
"134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host {"id": "6",
"name": "srv-3-comp.blabla.blub", "uuid":
"6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id": "1",
"name": "srv-2-comp.blabla.blub", "uuid":
"883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Found 4 hosts for allocation
after prioritization: [Host {"id": "7", "name": "srv-5-comp.blabla.blub",
"uuid": "8c753ce1-0253-49c9-9c7f-1f971a719ce6", "type"="Routing"}, Host
{"id": "5", "name": "srv-4-comp.blabla.blub", "uuid":
"134c625d-15ab-461a-a38d-36fb0b979b5d", "type"="Routing"}, Host {"id": "6",
"name": "srv-3-comp.blabla.blub", "uuid":
"6858713b-7e17-46d8-854d-24a13f9c7001", "type"="Routing"}, Host {"id": "1",
"name": "srv-2-comp.blabla.blub", "uuid":
"883f7f62-55f2-42f4-8242-16b6847c4204", "type"="Routing"}]
2022-08-29 10:51:56,233 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Looking for speed=500Mhz,
Ram=1024 MB
2022-08-29 10:51:56,233 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 7, name:
srv-5-comp.blabla.blub, uuid: 8c753ce1-0253-49c9-9c7f-1f971a719ce6} is KVM
hypervisor type, no max guest limit check needed
2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Host: 7 has cpu capability
(cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
2022-08-29 10:51:56,235 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 7 has enough
capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
cpuOverprovisioningFactor: 1.0
2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
and CPU after applying overprovisioning: 127200
2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 37100 , Requested CPU:
500
2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (176.37 GB)
189373743104 , Requested RAM: (1.00 GB) 1073741824
2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
available
2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
7, used: 90100, reserved: 0, actual total: 127200, total with
overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
,considerReservedCapacity?: true
2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
7, used: (73.90 GB) 79352037376, reserved: (0 bytes) 0, total: (250.27 GB)
268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
false , considerReservedCapacity?: true
2022-08-29 10:51:56,237 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
list: 7
2022-08-29 10:51:56,237 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 5, name:
srv-4-comp.blabla.blub, uuid: 134c625d-15ab-461a-a38d-36fb0b979b5d} is KVM
hypervisor type, no max guest limit check needed
2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Host: 5 has cpu capability
(cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
2022-08-29 10:51:56,239 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 5 has enough
capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
cpuOverprovisioningFactor: 1.0
2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
and CPU after applying overprovisioning: 127200
2022-08-29 10:51:56,240 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 116100 , Requested
CPU: 500
2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (233.77 GB)
251009040384 , Requested RAM: (1.00 GB) 1073741824
2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
available
2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
5, used: 11100, reserved: 0, actual total: 127200, total with
overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
,considerReservedCapacity?: true
2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
5, used: (16.50 GB) 17716740096, reserved: (0 bytes) 0, total: (250.27 GB)
268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
false , considerReservedCapacity?: true
2022-08-29 10:51:56,241 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
list: 5
2022-08-29 10:51:56,241 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 6, name:
srv-3-comp.blabla.blub, uuid: 6858713b-7e17-46d8-854d-24a13f9c7001} is KVM
hypervisor type, no max guest limit check needed
2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Host: 6 has cpu capability
(cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
2022-08-29 10:51:56,243 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 6 has enough
capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
cpuOverprovisioningFactor: 1.0
2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
and CPU after applying overprovisioning: 127200
2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 84300 , Requested CPU:
500
2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (190.02 GB)
204032835584 , Requested RAM: (1.00 GB) 1073741824
2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
available
2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
6, used: 42900, reserved: 0, actual total: 127200, total with
overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
,considerReservedCapacity?: true
2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
6, used: (60.25 GB) 64692944896, reserved: (0 bytes) 0, total: (250.27 GB)
268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
false , considerReservedCapacity?: true
2022-08-29 10:51:56,244 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
list: 6
2022-08-29 10:51:56,244 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Host {id: 1, name:
srv-2-comp.blabla.blub, uuid: 883f7f62-55f2-42f4-8242-16b6847c4204} is KVM
hypervisor type, no max guest limit check needed
2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Host: 1 has cpu capability
(cpu:48, speed:2650) to support requested CPU: 1 and requested speed: 500
2022-08-29 10:51:56,247 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Checking if host: 1 has enough
capacity for requested CPU: 500 and requested RAM: (1.00 GB) 1073741824 ,
cpuOverprovisioningFactor: 1.0
2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Hosts's actual total CPU: 127200
and CPU after applying overprovisioning: 127200
2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Free CPU: 79000 , Requested CPU:
500
2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Free RAM: (214.02 GB)
229802639360 , Requested RAM: (1.00 GB) 1073741824
2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Host has enough CPU and RAM
available
2022-08-29 10:51:56,248 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc CPU from host:
1, used: 48200, reserved: 0, actual total: 127200, total with
overprovisioning: 127200; requested cpu:500,alloc_from_last_host?:false
,considerReservedCapacity?: true
2022-08-29 10:51:56,249 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) STATS: Can alloc MEM from host:
1, used: (36.25 GB) 38923141120, reserved: (0 bytes) 0, total: (250.27 GB)
268725780480; requested mem: (1.00 GB) 1073741824, alloc_from_last_host?:
false , considerReservedCapacity?: true
2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Found a suitable host, adding to
list: 1
2022-08-29 10:51:56,249 DEBUG [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751
FirstFitRoutingAllocator) (logid:ba10fec5) Host Allocator returning 4
suitable hosts
2022-08-29 10:51:56,260 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
(logid:ba10fec5) VM instance {id: "83", name: "v-83-VM", uuid:
"b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} state
transited from [Starting] to [Stopped] with event [OperationFailed]. VM's
original host: null, new host: null, host before state transition: null
2022-08-29 10:51:56,260 WARN [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
(logid:ba10fec5) Unable to orchestrate start VM instance {id: "83", name:
"v-83-VM", uuid: "b1d1f9c6-eafe-4527-9335-e649c646aab0",
type="ConsoleProxy"} due to [Unable to create deployment, no usable volumes
found for the VM: 83].
com.cloud.utils.exception.CloudRuntimeException: Unable to create
deployment, no usable volumes found for the VM: 83
at
com.cloud.deploy.DeploymentPlanningManagerImpl.findSuitablePoolsForVolumes(DeploymentPlanningManagerImpl.java:1569)
at
com.cloud.deploy.DeploymentPlanningManagerImpl.checkClustersforDestination(DeploymentPlanningManagerImpl.java:1217)
at
com.cloud.deploy.DeploymentPlanningManagerImpl.planDeployment(DeploymentPlanningManagerImpl.java:527)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1126)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5315)
at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-08-29 10:51:56,262 DEBUG [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
(logid:ba10fec5) Done executing VM work job:
com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":83,"handlerName":"VirtualMachineManagerImpl"}
2022-08-29 10:51:56,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
(logid:ba10fec5) Complete async job-3539, jobStatus: FAILED, resultCode: 0,
result:
rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ANRVbmFibGUgdG8gb3JjaGVzdHJhdGUgc3RhcnQgVk0gaW5zdGFuY2Uge2lkOiAiODMiLCBuYW1lOiAidi04My1WTSIsIHV1aWQ6ICJiMWQxZjljNi1lYWZlLTQ1MjctOTMzNS1lNjQ5YzY0NmFhYjAiLCB0eXBlPSJDb25zb2xlUHJveHkifSBkdWUgdG8gW1VuYWJsZSB0byBjcmVhdGUgZGVwbG95bWVudCwgbm8gdXNhYmxlIHZvbHVtZXMgZm91bmQgZm9yIHRoZSBWTTogODNdLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAE3NyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAAUx3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgAMAP____9wdAAvamRrLmludGVybmFsLnJlZmxlY3QuR2VuZXJhdGVkTWV0aG9kQWNjZXNzb3I5NDVwdAAGaW52b2tlcHBzcQB-AAwCAAAAK3B0ADFqZGsuaW50ZXJuYWwucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABR0AAlqYXZhLmJhc2V0AAcxMS4wLjE2c3EAfgAMAgAAAjZwdAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AFHEAfgAYcQB-ABlzcQB-AAwBAAAAa3EAfgAOdAAiY29tLmNsb3VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZXJQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9icHBzcQB-AAwBAAAVP3EAfgAOcQB-AA9xAH4AEHEAfgAgcHBzcQB-AAwBAAAAZnEAfgAOdAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JwcHNxAH4ADAEAAAJscQB-AA50AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRwcHNxAH4ADAEAAAAwcQB-AA50AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnBwc3EAfgAMAQAAADdxAH4ADnQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHBwc3EAfgAMAQAAAGZxAH4ADnQAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AMHQAD2NhbGxXaXRoQ29udGV4dHBwc3EAfgAMAQAAADRxAH4ADnEAfgAzcQB-ADB0AA5ydW5XaXRoQ29udGV4dHBwc3EAfgAMAQAAAC1xAH4ADnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAscQB-AC1wcHNxAH4ADAEAAAI4cQB-AA5xAH4AJ3EAfgAocQB-AC1wcHNxAH4ADAIAAAIDcHQALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAxcQB-ABhxAH4AGXNxAH4ADAIAAAEIcHQAH2phdmEudXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAAEaHB0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnEAfgAYcQB-ABlzcQB-AAwCAAACdHB0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-AEJxAH4ALXEAfgAYcQB-ABlzcQB-AAwCAAADPXB0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4ALXEAfgAYcQB-ABlzcgAfamF2YS51dGlsLkNvbGxlY3Rpb25zJEVtcHR5TGlzdHq4F7Q8p57eAgAAeHB4AAAQmncIAAAAAAAAAAB4
2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
(logid:ba10fec5) Publish async job-3539 complete on message bus
2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
(logid:ba10fec5) Wake up jobs related to job-3539
2022-08-29 10:51:56,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
(logid:ba10fec5) Update db status for job-3539
2022-08-29 10:51:56,266 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539 ctx-073f2751)
(logid:ba10fec5) Wake up jobs joined with job-3539 and disjoin all subjobs
created from job- 3539
2022-08-29 10:51:56,276 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5) Done
with run of VM work job: com.cloud.vm.VmWorkStart for VM 83, job origin:
3445
2022-08-29 10:51:56,276 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5) Done
executing com.cloud.vm.VmWorkStart for job-3539
2022-08-29 10:51:56,278 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-58:ctx-dd8fe695 job-3445/job-3539) (logid:ba10fec5)
Remove job-3539 from job monitoring
2022-08-29 10:51:56,285 WARN [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Unable to allocate console
proxy standby capacity for zone [1] due to [Unable to orchestrate start VM
instance {id: "83", name: "v-83-VM", uuid:
"b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to [Unable
to create deployment, no usable volumes found for the VM: 83].].
com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate
start VM instance {id: "83", name: "v-83-VM", uuid:
"b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to [Unable
to create deployment, no usable volumes found for the VM: 83].
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5319)
at jdk.internal.reflect.GeneratedMethodAccessor945.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5439)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-08-29 10:51:56,287 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
(consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) received console proxy alert
2022-08-29 10:51:56,288 DEBUG [c.c.a.ConsoleProxyAlertAdapter]
(consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) Console proxy creation
failure, zone: blabla
2022-08-29 10:51:56,289 WARN [c.c.a.AlertManagerImpl]
(consoleproxy-1:ctx-88c25b3a) (logid:d39faaa7) alertType=[10]
dataCenterId=[1] podId=[null] clusterId=[null] message=[Console proxy
creation failure. zone: blabla, error details: Unable to orchestrate start
VM instance {id: "83", name: "v-83-VM", uuid:
"b1d1f9c6-eafe-4527-9335-e649c646aab0", type="ConsoleProxy"} due to [Unable
to create deployment, no usable volumes found for the VM: 83].].
Am Mo., 29. Aug. 2022 um 09:40 Uhr schrieb Slavka Peleva
<sl...@storpool.com.invalid>:
> Hi Chris,
>
> Did you recreate the system VMs? In the 4.17 version, the `systemvm.iso` is
> deprecated. In its place, you should have the file on agents
> `/usr/share/cloudstack-common/vms/agent.zip`.
> Can you share the complete log if the system VMs are newly created after
> the upgrade?
>
> Best regards,
> Slavka
>
> On Sun, Aug 28, 2022 at 2:16 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
>
> > Hi everyone,
> >
> > faceing some challanges again after upgradeing to 4.17.
> > Did as explained in the docs.
> >
> > Afterwards I am now not able to migrate system VMs to different hosts.
> > Getting the error
> > "Exception during migrate: org.libvirt.LibvirtException: Cannot access
> > storage file '/usr/share/cloudstack-common/vms/systemvm.iso': No such
> file
> > or directory"
> >
> > Checked the exitstence of the file on the servers:
> > Mangement-Server has a "systemvm.iso.bak"
> > Hosts - no "systemvm.iso" at all present.
> >
> > Maybe someone has an idea on how to move on with this?
> >
> > Regards,
> > Chris
> >
>
Re: Error migrate System VMs after upgrageing to 4.17
Posted by Slavka Peleva <sl...@storpool.com.INVALID>.
Hi Chris,
Did you recreate the system VMs? In the 4.17 version, the `systemvm.iso` is
deprecated. In its place, you should have the file on agents
`/usr/share/cloudstack-common/vms/agent.zip`.
Can you share the complete log if the system VMs are newly created after
the upgrade?
Best regards,
Slavka
On Sun, Aug 28, 2022 at 2:16 PM Vash_X@gmx.de <Va...@gmx.de> wrote:
> Hi everyone,
>
> faceing some challanges again after upgradeing to 4.17.
> Did as explained in the docs.
>
> Afterwards I am now not able to migrate system VMs to different hosts.
> Getting the error
> "Exception during migrate: org.libvirt.LibvirtException: Cannot access
> storage file '/usr/share/cloudstack-common/vms/systemvm.iso': No such file
> or directory"
>
> Checked the exitstence of the file on the servers:
> Mangement-Server has a "systemvm.iso.bak"
> Hosts - no "systemvm.iso" at all present.
>
> Maybe someone has an idea on how to move on with this?
>
> Regards,
> Chris
>