You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "midhunpjos (via GitHub)" <gi...@apache.org> on 2025/02/06 13:25:06 UTC
[D] Cloudstack Agent Java IO exception [cloudstack]
GitHub user midhunpjos created a discussion: Cloudstack Agent Java IO exception
### problem
Network Actions,VM Deployments,router health checks are failing.
On cloudstack agent logs below is printed
2025-02-06 09:14:09,343 WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:[]) (logid:64df14bf) Exception [Cannot run program "/usr/share/
cloudstack-common/scripts/network/domr/router_proxy.sh": error=0, Failed to exec spawn helper: pid: 605941, exit value: 1] occurred when attempting to run command [/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh filesystem_writable_check.py 169.254.156.203 ]. java.io.IOException: Cannot run prog
ram "/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh": error=0, Failed to exec spawn helper: pid: 605941, exit value: 1
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)
at com.cloud.utils.script.Script.execute(Script.java:232)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:553)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:541)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.checkRouterFileSystem(VirtualRoutingResource.java:419)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.execute(VirtualRoutingResource.java:391)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeQueryCommand(VirtualRoutingResource.java:227)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:129)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:35)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:29)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1945)
at com.cloud.agent.Agent.processRequest(Agent.java:686)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1109)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.io.IOException: error=0, Failed to exec spawn helper: pid: 605941, exit value: 1
at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:314)
at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:244)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)
... 13 more
2025-02-06 09:14:46,697 WARN [kvm.storage.MultipathSCSIAdapterBase] (MultipathMapCleanupJob:[]) (logid:) Exception [Cannot run program "/usr/share/cloudstac
k-common/scripts/storage/multipath/cleanStaleMaps.sh": error=0, Failed to exec spawn helper: pid: 605943, exit value: 1] occurred when attempting to run command [/usr/share/cloudstack-common/scripts/storage/multipath/cleanStaleMaps.sh ]. java.io.IOException: Cannot run program "/usr/share/cloudstack-common/script
s/storage/multipath/cleanStaleMaps.sh": error=0, Failed to exec spawn helper: pid: 605943, exit value: 1
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)
at com.cloud.utils.script.Script.execute(Script.java:232)
at com.cloud.hypervisor.kvm.storage.MultipathSCSIAdapterBase.runScript(MultipathSCSIAdapterBase.java:382)
at com.cloud.hypervisor.kvm.storage.MultipathSCSIAdapterBase.cleanupStaleMaps(MultipathSCSIAdapterBase.java:572)
at com.cloud.hypervisor.kvm.storage.MultipathSCSIAdapterBase$1.run(MultipathSCSIAdapterBase.java:122)
at java.base/java.util.TimerThread.mainLoop(Timer.java:566)
at java.base/java.util.TimerThread.run(Timer.java:516)
Caused by: java.io.IOException: error=0, Failed to exec spawn helper: pid: 605943, exit value: 1
at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:314)
at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:244)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)
... 7 more
### versions
Cloudstack 4.20 + KVM
CEPH as Primary Storage
### The steps to reproduce the bug
1.Deploy Instance, add firewall rules for guest networks are failing
2.
3.
...
### What to do about it?
Restarting Cloudstack agent resolved the issue
GitHub link: https://github.com/apache/cloudstack/discussions/10336
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Cloudstack Agent Java IO exception [cloudstack]
Posted by "midhunpjos (via GitHub)" <gi...@apache.org>.
GitHub user midhunpjos added a comment to the discussion: Cloudstack Agent Java IO exception
> googled the error , it looks like a config is needed
>
> ```
> -Djdk.lang.Process.launchMechanism=vfork
> ```
>
> maybe add to `JAVA_OPTS` in `/etc/default/cloudstack-agent`
Noted @weizhouapache. Will relay on this if the issue is happening again
GitHub link: https://github.com/apache/cloudstack/discussions/10336#discussioncomment-12085587
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Cloudstack Agent Java IO exception [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
GitHub user DaanHoogland added a comment to the discussion: Cloudstack Agent Java IO exception
@midhunpjos , it looks like the agent is not allowed to write on disk. If so only restarting the agent would not solve the issue on NFS. I do not know what the behaviour on Ceph is. Can you reliably reproduce this?
GitHub link: https://github.com/apache/cloudstack/discussions/10336#discussioncomment-12081921
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Cloudstack Agent Java IO exception [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache edited a comment on the discussion: Cloudstack Agent Java IO exception
googled the error , it looks like a config is needed
```
-Djdk.lang.Process.launchMechanism=vfork
```
maybe add to `JAVA_OPTS` in `/etc/default/cloudstack-agent`
GitHub link: https://github.com/apache/cloudstack/discussions/10336#discussioncomment-12082460
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Cloudstack Agent Java IO exception [cloudstack]
Posted by "PPisz (via GitHub)" <gi...@apache.org>.
GitHub user PPisz added a comment to the discussion: Cloudstack Agent Java IO exception
We have the same problem regularly, today it occurred for about 1 hour on all hosts and disappeared by itself. The environment is Ubuntu 22, CS 4.19.3 - we will test the solution given in the comment. Despite deep environment checks we were not able to pinpoint what is happening. During this problem you cannot ssh to VR via cloud0 (we have a simple linux bridge + vlan configuration). Not all VRs have the issue at the same time, restarting or migrating the VR to another host does not solve the issue.
GitHub link: https://github.com/apache/cloudstack/discussions/10336#discussioncomment-12094483
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Cloudstack Agent Java IO exception [cloudstack]
Posted by "midhunpjos (via GitHub)" <gi...@apache.org>.
GitHub user midhunpjos closed a discussion: Cloudstack Agent Java IO exception
### problem
Network Actions,VM Deployments,router health checks are failing.
On cloudstack agent logs below is printed
2025-02-06 09:14:09,343 WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:[]) (logid:64df14bf) Exception [Cannot run program "/usr/share/
cloudstack-common/scripts/network/domr/router_proxy.sh": error=0, Failed to exec spawn helper: pid: 605941, exit value: 1] occurred when attempting to run command [/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh filesystem_writable_check.py 169.254.156.203 ]. java.io.IOException: Cannot run prog
ram "/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh": error=0, Failed to exec spawn helper: pid: 605941, exit value: 1
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)
at com.cloud.utils.script.Script.execute(Script.java:232)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:553)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:541)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.checkRouterFileSystem(VirtualRoutingResource.java:419)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.execute(VirtualRoutingResource.java:391)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeQueryCommand(VirtualRoutingResource.java:227)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:129)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:35)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:29)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1945)
at com.cloud.agent.Agent.processRequest(Agent.java:686)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1109)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.io.IOException: error=0, Failed to exec spawn helper: pid: 605941, exit value: 1
at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:314)
at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:244)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)
... 13 more
2025-02-06 09:14:46,697 WARN [kvm.storage.MultipathSCSIAdapterBase] (MultipathMapCleanupJob:[]) (logid:) Exception [Cannot run program "/usr/share/cloudstac
k-common/scripts/storage/multipath/cleanStaleMaps.sh": error=0, Failed to exec spawn helper: pid: 605943, exit value: 1] occurred when attempting to run command [/usr/share/cloudstack-common/scripts/storage/multipath/cleanStaleMaps.sh ]. java.io.IOException: Cannot run program "/usr/share/cloudstack-common/script
s/storage/multipath/cleanStaleMaps.sh": error=0, Failed to exec spawn helper: pid: 605943, exit value: 1
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)
at com.cloud.utils.script.Script.execute(Script.java:232)
at com.cloud.hypervisor.kvm.storage.MultipathSCSIAdapterBase.runScript(MultipathSCSIAdapterBase.java:382)
at com.cloud.hypervisor.kvm.storage.MultipathSCSIAdapterBase.cleanupStaleMaps(MultipathSCSIAdapterBase.java:572)
at com.cloud.hypervisor.kvm.storage.MultipathSCSIAdapterBase$1.run(MultipathSCSIAdapterBase.java:122)
at java.base/java.util.TimerThread.mainLoop(Timer.java:566)
at java.base/java.util.TimerThread.run(Timer.java:516)
Caused by: java.io.IOException: error=0, Failed to exec spawn helper: pid: 605943, exit value: 1
at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:314)
at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:244)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)
... 7 more
### versions
Cloudstack 4.20 + KVM
CEPH as Primary Storage
### The steps to reproduce the bug
1.Deploy Instance, add firewall rules for guest networks are failing
2.
3.
...
### What to do about it?
Restarting Cloudstack agent resolved the issue
GitHub link: https://github.com/apache/cloudstack/discussions/10336
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Cloudstack Agent Java IO exception [cloudstack]
Posted by "midhunpjos (via GitHub)" <gi...@apache.org>.
GitHub user midhunpjos added a comment to the discussion: Cloudstack Agent Java IO exception
@weizhouapache Seems like unattended-upgrades are causing this and after every upgrade we need the agent restart
ref: https://bugs.launchpad.net/ubuntu/+source/openjdk-17/+bug/2055280
From my server logs I could see the JRE being upgraded frequently causing this issue.
Disabling unattended-upgrades can be a workaround
GitHub link: https://github.com/apache/cloudstack/discussions/10336#discussioncomment-12114145
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Cloudstack Agent Java IO exception [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: Cloudstack Agent Java IO exception
good finding @midhunpjos
GitHub link: https://github.com/apache/cloudstack/discussions/10336#discussioncomment-12114482
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Cloudstack Agent Java IO exception [cloudstack]
Posted by "PPisz (via GitHub)" <gi...@apache.org>.
GitHub user PPisz added a comment to the discussion: Cloudstack Agent Java IO exception
After upgrading to 4.20 (which we rolled back from) we also had the bug described above.
GitHub link: https://github.com/apache/cloudstack/discussions/10336#discussioncomment-12094562
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Cloudstack Agent Java IO exception [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
GitHub user DaanHoogland added a comment to the discussion: Cloudstack Agent Java IO exception
I understand @midhunpjos , however for anybody to investigate the issue we must be able to replay it . You may be able to find more information in older log-entries. It is important to know how the read-only state of the agent file system came to be. If not the work around you proivided is all we have. (i.e. restart the agent)
GitHub link: https://github.com/apache/cloudstack/discussions/10336#discussioncomment-12081952
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Cloudstack Agent Java IO exception [cloudstack]
Posted by "midhunpjos (via GitHub)" <gi...@apache.org>.
GitHub user midhunpjos added a comment to the discussion: Cloudstack Agent Java IO exception
@DaanHoogland It happened to all Hosts in a cluster and just restarting agent everything was back to normal in all hosts.
GitHub link: https://github.com/apache/cloudstack/discussions/10336#discussioncomment-12081922
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org
Re: [D] Cloudstack Agent Java IO exception [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
GitHub user weizhouapache added a comment to the discussion: Cloudstack Agent Java IO exception
googled the error , it looks like a config is needed
```
-Djdk.lang.Process.launchMechanism=vfork
```
GitHub link: https://github.com/apache/cloudstack/discussions/10336#discussioncomment-12082460
----
This is an automatically sent email for users@cloudstack.apache.org.
To unsubscribe, please send an email to: users-unsubscribe@cloudstack.apache.org