You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2021/03/25 09:33:36 UTC
[GitHub] [cloudstack] ustcweizhou opened a new pull request #4869: VR: fix rsyslog compresses log files but not release disk space in VR
ustcweizhou opened a new pull request #4869:
URL: https://github.com/apache/cloudstack/pull/4869
### Description
We had critical issue with VR recently. The VRs of shared network or vpc stops working after some days.
After investigation, I found that the disk space is full
```
root@r-10-VM:~# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/vda5 2086316 2069932 0 100% /
```
logrotate/ryslog has compresses the log files, but space is not released. see `lsof |grep deleted`
```
root@r-10-VM:~# lsof |grep deleted
rsyslogd 960 root 12w REG 254,5 493060096 137 /var/log/daemon.log.1 (deleted)
rsyslogd 960 root 13w REG 254,5 17715200 110 /var/log/messages.1 (deleted)
rsyslogd 960 root 16w REG 254,5 545968128 342 /var/log/auth.log.1 (deleted)
rsyslogd 960 root 18w REG 254,5 38313984 341 /var/log/cron.log.1 (deleted)
rsyslogd 960 962 in:imuxso root 12w REG 254,5 493060096 137 /var/log/daemon.log.1 (deleted)
rsyslogd 960 962 in:imuxso root 13w REG 254,5 17715200 110 /var/log/messages.1 (deleted)
rsyslogd 960 962 in:imuxso root 16w REG 254,5 545968128 342 /var/log/auth.log.1 (deleted)
rsyslogd 960 962 in:imuxso root 18w REG 254,5 38313984 341 /var/log/cron.log.1 (deleted)
rsyslogd 960 963 in:imklog root 12w REG 254,5 493060096 137 /var/log/daemon.log.1 (deleted)
rsyslogd 960 963 in:imklog root 13w REG 254,5 17715200 110 /var/log/messages.1 (deleted)
rsyslogd 960 963 in:imklog root 16w REG 254,5 545968128 342 /var/log/auth.log.1 (deleted)
rsyslogd 960 963 in:imklog root 18w REG 254,5 38313984 341 /var/log/cron.log.1 (deleted)
rsyslogd 960 964 in:imfile root 12w REG 254,5 493060096 137 /var/log/daemon.log.1 (deleted)
rsyslogd 960 964 in:imfile root 13w REG 254,5 17715200 110 /var/log/messages.1 (deleted)
rsyslogd 960 964 in:imfile root 16w REG 254,5 545968128 342 /var/log/auth.log.1 (deleted)
rsyslogd 960 964 in:imfile root 18w REG 254,5 38313984 341 /var/log/cron.log.1 (deleted)
rsyslogd 960 965 in:imudp root 12w REG 254,5 493060096 137 /var/log/daemon.log.1 (deleted)
rsyslogd 960 965 in:imudp root 13w REG 254,5 17715200 110 /var/log/messages.1 (deleted)
rsyslogd 960 965 in:imudp root 16w REG 254,5 545968128 342 /var/log/auth.log.1 (deleted)
rsyslogd 960 965 in:imudp root 18w REG 254,5 38313984 341 /var/log/cron.log.1 (deleted)
rsyslogd 960 966 rs:main root 12w REG 254,5 493060096 137 /var/log/daemon.log.1 (deleted)
rsyslogd 960 966 rs:main root 13w REG 254,5 17715200 110 /var/log/messages.1 (deleted)
rsyslogd 960 966 rs:main root 16w REG 254,5 545968128 342 /var/log/auth.log.1 (deleted)
rsyslogd 960 966 rs:main root 18w REG 254,5 38313984 341 /var/log/cron.log.1 (deleted)
```
workaround: restarting rsyslog to release the space.
```
systemctl restart rsyslog
```
The root cause is, the following command does not work in 4.15 template
```
root@r-10-VM:~# invoke-rc.d rsyslog rotate
[FAIL] Closing open files: rsyslogd failed!
```
Fix: use `/usr/lib/rsyslog/rsyslog-rotate` instead
```
root@r-10-VM:~# /usr/lib/rsyslog/rsyslog-rotate
root@r-10-VM:~# cat /usr/lib/rsyslog/rsyslog-rotate
if [ -d /run/systemd/system ]; then
systemctl kill -s HUP rsyslog.service
else
invoke-rc.d rsyslog rotate > /dev/null
fi
```
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [X] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
### Feature/Enhancement Scale or Bug Severity
#### Bug Severity
- [ ] BLOCKER
- [X] Critical
- [ ] Major
- [ ] Minor
- [ ] Trivial
### Screenshots (if appropriate):
### How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
<!-- Please read the [CONTRIBUTING](https://github.com/apache/cloudstack/blob/master/CONTRIBUTING.md) document -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4869: VR: fix rsyslog compresses log files but not release disk space in VR
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4869:
URL: https://github.com/apache/cloudstack/pull/4869#issuecomment-809323542
Packaging result: :heavy_check_mark: centos7 :heavy_check_mark: centos8 :heavy_check_mark: debian. SL-JID 279
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4869: VR: fix rsyslog compresses log files but not release disk space in VR
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4869:
URL: https://github.com/apache/cloudstack/pull/4869#issuecomment-810040991
@blueorangutan test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4869: VR: fix rsyslog compresses log files but not release disk space in VR
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4869:
URL: https://github.com/apache/cloudstack/pull/4869#issuecomment-809341144
@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd merged pull request #4869: VR: fix rsyslog compresses log files but not release disk space in VR
Posted by GitBox <gi...@apache.org>.
rhtyd merged pull request #4869:
URL: https://github.com/apache/cloudstack/pull/4869
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4869: VR: fix rsyslog compresses log files but not release disk space in VR
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4869:
URL: https://github.com/apache/cloudstack/pull/4869#issuecomment-809881572
<b>Trillian test result (tid-290)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 49841 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4869-t290-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 81 look OK, 5 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
ContextSuite context=TestKubernetesCluster>:teardown | `Error` | 71.94 | test_kubernetes_clusters.py
test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true | `Failure` | 306.55 | test_routers_network_ops.py
test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false | `Failure` | 305.44 | test_routers_network_ops.py
test_01_migrate_VM_and_root_volume | `Error` | 70.24 | test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks | `Error` | 49.06 | test_vm_life_cycle.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | `Failure` | 492.69 | test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers | `Failure` | 483.83 | test_vpc_redundant.py
test_05_rvpc_multi_tiers | `Failure` | 392.17 | test_vpc_redundant.py
test_05_rvpc_multi_tiers | `Error` | 418.08 | test_vpc_redundant.py
test_hostha_kvm_host_fencing | `Error` | 651.60 | test_hostha_kvm.py
test_hostha_kvm_host_recovering | `Error` | 664.73 | test_hostha_kvm.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4869: VR: fix rsyslog compresses log files but not release disk space in VR
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4869:
URL: https://github.com/apache/cloudstack/pull/4869#issuecomment-809286004
Yes @ustcweizhou we've plan for 4.15.1.0 already in effect, see https://markmail.org/message/deicvofukqom2w5u
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4869: VR: fix rsyslog compresses log files but not release disk space in VR
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4869:
URL: https://github.com/apache/cloudstack/pull/4869#issuecomment-810041408
@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4869: VR: fix rsyslog compresses log files but not release disk space in VR
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4869:
URL: https://github.com/apache/cloudstack/pull/4869#issuecomment-810619140
<b>Trillian test result (tid-303)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 47268 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4869-t303-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 82 look OK, 4 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
ContextSuite context=TestKubernetesCluster>:teardown | `Error` | 72.10 | test_kubernetes_clusters.py
test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true | `Failure` | 303.47 | test_routers_network_ops.py
test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false | `Failure` | 303.72 | test_routers_network_ops.py
test_01_migrate_VM_and_root_volume | `Error` | 67.21 | test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks | `Error` | 50.09 | test_vm_life_cycle.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | `Failure` | 492.55 | test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers | `Failure` | 428.49 | test_vpc_redundant.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4869: VR: fix rsyslog compresses log files but not release disk space in VR
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4869:
URL: https://github.com/apache/cloudstack/pull/4869#issuecomment-809340449
@blueorangutan test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] ustcweizhou commented on pull request #4869: VR: fix rsyslog compresses log files but not release disk space in VR
Posted by GitBox <gi...@apache.org>.
ustcweizhou commented on pull request #4869:
URL: https://github.com/apache/cloudstack/pull/4869#issuecomment-806503440
@rhtyd @DaanHoogland
this is a critical issue. not sure if the issue exists in 4.14.
by the way, do we have plan for 4.15.1.0 ? or 4.15.0.1 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4869: VR: fix rsyslog compresses log files but not release disk space in VR
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4869:
URL: https://github.com/apache/cloudstack/pull/4869#issuecomment-809286207
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org