You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "vdombrovski (via GitHub)" <gi...@apache.org> on 2023/06/19 14:31:39 UTC
[GitHub] [cloudstack] vdombrovski opened a new issue, #7576: Logrotate issue in VirtualRouter
vdombrovski opened a new issue, #7576:
URL: https://github.com/apache/cloudstack/issues/7576
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and main branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
~~~
VR
~~~
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on main branch.
-->
~~~
4.17.2
~~~
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network, advanced networking, etc. N/A otherwise
-->
Advanced networking, KVM based, VR using the official 4.17 System VM image (Debian GNU/Linux 5.0 (64-bit))
##### OS / ENVIRONMENT
<!--
Information about the environment if relevant, N/A otherwise
-->
N/A
##### SUMMARY
<!-- Explain the problem/feature briefly -->
Haproxy log files in virtual router are not released by rsyslogd, causing a full in VR
```
root@r-5033-VM:~# df -h
[...]
/dev/vda6 4.2G 4.2G 0 100% /
lsof | grep deleted
rsyslogd 342 root 16w REG 254,6 3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 346 in:imuxso root 16w REG 254,6 3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 347 in:imklog root 16w REG 254,6 3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 348 imudp(w0) root 16w REG 254,6 3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 349 rs:main root 16w REG 254,6 3142168576 128005 /var/log/haproxy.log.1 (deleted)
/etc/init.d/rsyslog restart
df -h
[...]
/dev/vda6 4.2G 1.3G 3.0G 30% /
```
##### STEPS TO REPRODUCE
<!--
For bugs, show exactly how to reproduce the problem, using a minimal test-case. Use Screenshots if accurate.
For new features, show how the feature would be used.
-->
<!-- Paste example playbooks or commands between quotes below -->
~~~
- Deploy a VR
- Add a load-balancing rule to a public IP handled by this VR
- Wait :)
~~~
<!-- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS
<!-- What did you expect to happen when running the steps above? -->
~~~
Old log files get released
~~~
##### ACTUAL RESULTS
<!-- What actually happened? -->
<!-- Paste verbatim command output between quotes below -->
~~~
lsof | grep deleted
rsyslogd 342 root 16w REG 254,6 3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 346 in:imuxso root 16w REG 254,6 3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 347 in:imklog root 16w REG 254,6 3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 348 imudp(w0) root 16w REG 254,6 3142168576 128005 /var/log/haproxy.log.1 (deleted)
rsyslogd 342 349 rs:main root 16w REG 254,6 3142168576 128005 /var/log/haproxy.log.1 (deleted)
~~~
Deleted file is held, causing a full on the VR, causing all the deployments inside the network to fail with
```
WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-2:null) (logid:9b4056c5) Failed to create file /var/cache/cloud/monitor_service.json.10cd2d0e-fd3c-
4bc8-9868-7041cdaad464 in VR 169.254.154.26
java.io.IOException: Error during SCP transfer.
at com.trilead.ssh2.SCPClient.put(SCPClient.java:518)
at com.trilead.ssh2.SCPClient.put(SCPClient.java:471)
at com.cloud.utils.ssh.SshHelper.scpTo(SshHelper.java:191)
at com.cloud.utils.ssh.SshHelper.scpTo(SshHelper.java:64)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createFileInVR(LibvirtComputingResource.java:483)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfigToVR(VirtualRoutingResource.java:226)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.applyConfig(VirtualRoutingResource.java:244)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:150)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:35)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:29)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1831)
at com.cloud.agent.Agent.processRequest(Agent.java:660)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1080)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: Remote scp terminated with error (scp: /var/cache/cloud//monitor_service.json.10cd2d0e-fd3c-4bc8-9868-7041cdaad464: No space left on device).
at com.trilead.ssh2.SCPClient.readResponse(SCPClient.java:51)
at com.trilead.ssh2.SCPClient.sendBytes(SCPClient.java:142)
at com.trilead.ssh2.SCPClient.put(SCPClient.java:514)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] vdombrovski commented on issue #7576: Logrotate issue in VirtualRouter
Posted by "vdombrovski (via GitHub)" <gi...@apache.org>.
vdombrovski commented on issue #7576:
URL: https://github.com/apache/cloudstack/issues/7576#issuecomment-1597287467
> > Hi Do you need something to merge the PR ?
>
> @bragonznx Based on PR merge policy, we need (1) at least 2 * approvals (including 1 manual validation); (2) all trillian tests pass.
@weizhouapache the trillian tests look broken, judging by the logs, the error they present has nothing to do with the change made in this PR. What needs to be done to get this PR merged then? Do we need to wait for someone to fix the broken tests first?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache closed issue #7576: Logrotate issue in VirtualRouter
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache closed issue #7576: Logrotate issue in VirtualRouter
URL: https://github.com/apache/cloudstack/issues/7576
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache closed issue #7576: Logrotate issue in VirtualRouter
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache closed issue #7576: Logrotate issue in VirtualRouter
URL: https://github.com/apache/cloudstack/issues/7576
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] bragonznx commented on issue #7576: Logrotate issue in VirtualRouter
Posted by "bragonznx (via GitHub)" <gi...@apache.org>.
bragonznx commented on issue #7576:
URL: https://github.com/apache/cloudstack/issues/7576#issuecomment-1597164654
Hi
Do you need something to merge the PR ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] bragonznx commented on issue #7576: Logrotate issue in VirtualRouter
Posted by "bragonznx (via GitHub)" <gi...@apache.org>.
bragonznx commented on issue #7576:
URL: https://github.com/apache/cloudstack/issues/7576#issuecomment-1597440860
Good to see that we are moving forward :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #7576: Logrotate issue in VirtualRouter
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #7576:
URL: https://github.com/apache/cloudstack/issues/7576#issuecomment-1573305397
added to 4.18.1.0 milestone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #7576: Logrotate issue in VirtualRouter
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #7576:
URL: https://github.com/apache/cloudstack/issues/7576#issuecomment-1597180097
> Hi Do you need something to merge the PR ?
@bragonznx
Based on PR merge policy, we need (1) at least 2 * approvals (including 1 manual validation); (2) all trillian tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #7576: Logrotate issue in VirtualRouter
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #7576:
URL: https://github.com/apache/cloudstack/issues/7576#issuecomment-1597297792
> > > Hi Do you need something to merge the PR ?
> >
> >
> > @bragonznx Based on PR merge policy, we need (1) at least 2 * approvals (including 1 manual validation); (2) all trillian tests pass.
>
> @weizhouapache the trillian tests look broken, judging by the logs, the error they present has nothing to do with the change made in this PR. What needs to be done to get this PR merged then? Do we need to wait for someone to fix the broken tests first?
@vdombrovski
I kicked a trillian test just now. Let's wait
Also we need someone else to review and test it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] vdombrovski commented on issue #7576: Logrotate issue in VirtualRouter
Posted by "vdombrovski (via GitHub)" <gi...@apache.org>.
vdombrovski commented on issue #7576:
URL: https://github.com/apache/cloudstack/issues/7576#issuecomment-1573280620
@rajujith The VR is actually running Debian 11:
```
cat /etc/*release*
Cloudstack Release 4.17.2 Fri 09 Dec 2022 12:51:18 PM UTC
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
```
I've said previously "Debian 5", because ACS doesn't properly recognize it (which is another bug in itself, but out of scope for this issue):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rajujith commented on issue #7576: Logrotate issue in VirtualRouter
Posted by "rajujith (via GitHub)" <gi...@apache.org>.
rajujith commented on issue #7576:
URL: https://github.com/apache/cloudstack/issues/7576#issuecomment-1573263674
Hi, @vdombrovski the SystemVM template for 4.17.2 should be a newer one, not a Debian 5 based one. Could you check whether you should upgrade the systemVM template? that could help.
http://docs.cloudstack.apache.org/en/4.17.2.0/upgrading/upgrade/upgrade-4.17.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #7576: Logrotate issue in VirtualRouter
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #7576:
URL: https://github.com/apache/cloudstack/issues/7576#issuecomment-1598274676
fixed via #7587
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org