You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Cloud List <cl...@sg.or.id> on 2016/11/28 03:46:52 UTC
ACS 4.8.1: Virtual router - /var/log always full
Dear all,
After upgrading to ACS 4.8.1, one of our virtual router's /var/log
partition is always full and used up quite fast. This caused the VR not
able to serve DHCP and password requests from VM.
root@r-4155-VM:/var/log# df -h
Filesystem Size Used Avail
Use% Mounted on
rootfs 461M 158M 280M
37% /
udev 10M 0 10M
0% /dev
tmpfs 25M 236K 25M
1% /run
/dev/disk/by-uuid/30c81d3d-ee9f-4a88-81c1-5f349b22ba1d 461M 158M 280M
37% /
tmpfs 5.0M 0 5.0M
0% /run/lock
tmpfs 157M 0 157M
0% /run/shm
/dev/vda1 73M 23M 47M
33% /boot
/dev/vda6 92M 5.6M 81M
7% /home
/dev/vda8 184M 6.2M 169M
4% /opt
/dev/vda11 92M 5.6M 81M
7% /tmp
/dev/vda7 751M 493M 219M
70% /usr
/dev/vda9 563M 282M 252M
53% /var
/dev/vda10 184M 176M 0
100% /var/log
Even after rotating and clearing the logs, the usage of /var/log is only
4.7M so I am not too sure where is the 176M coming from.
root@r-4155-VM:/var/log# du -h
1.0K ./samba
3.8M ./sysstat
68K ./apt
7.0K ./apache2
3.0K ./fsck
317K ./installer/cdebconf
809K ./installer
1.0K ./news
12K ./lost+found
1.0K ./ntpstats
4.7M .
/dev/vda10 184M 175M 475K
100% /var/log
I would need to clear the logs and do a "service dnsmasq restart" regularly
to make the VR functioning again, which is quite troublesome.
Any advice is greatly appreciated.
Looking forward to your reply, thank you.
Cheers.
Re: ACS 4.8.1: Virtual router - /var/log always full
Posted by Cloud List <cl...@sg.or.id>.
Thanks Wei for your tips, greatly appreciated. Will do what you have
suggested.
Hi Will, we are using KVM, so not too sure if it's related to the bug you
just found.
Thank you.
On Mon, Nov 28, 2016 at 2:02 PM, Wei ZHOU <us...@gmail.com> wrote:
> This is because the free space in systemvm template is not big, and
> logrotate rules are not wise enough to hold the logs.
>
> you can remove the "delaycompress" and reduce the "size" in
> systemvm/patches/debian/config/etc/logrotate.d/rsyslog
> (/etc/logrotate.d/rsyslog in VR/systemvms), then restart logrotate to fix
> it.
>
> -Wei
>
> 2016-11-28 5:45 GMT+01:00 Will Stevens <ws...@cloudops.com>:
>
> > Are you using XenServer? We recently found a bug in XenServer which has
> > similar characteristics. I can put together details of the problem if
> > yes...
> >
> > Cheers,
> >
> > Will
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Sun, Nov 27, 2016 at 11:36 PM, Cloud List <cl...@sg.or.id>
> wrote:
> >
> > > Dear all,
> > >
> > > Please ignore this, managed to find the reason - paswd_server_ip
> process
> > is
> > > holding to a log file which has been deleted earlier to fix the disk
> > space
> > > issue. Restarted the process and the disk space is fine now.
> > >
> > > ===
> > > passwd_se 2199 root 1w REG 254,10
> > > 171802407 15 /var/log/cloud.log (deleted)
> > > passwd_se 2199 root 2w REG 254,10
> > > 171802407 15 /var/log/cloud.log (deleted)
> > > passwd_se 2199 root 3w REG 254,10
> > > 171802407 15 /var/log/cloud.log (deleted)
> > > python 2202 root 3w REG 254,10
> > > 171802407 15 /var/log/cloud.log (deleted)
> > >
> > > 2199 ? S 0:00 /bin/bash /opt/cloud/bin/passwd_server_ip
> > > X.X.X.2 dummy
> > > 2202 ? S 7:13 python /opt/cloud/bin/passwd_server_ip.py
> > > X.X.X.2
> > > ===
> > >
> > > Thank you.
> > >
> > >
> > > On Mon, Nov 28, 2016 at 11:46 AM, Cloud List <cl...@sg.or.id>
> > wrote:
> > >
> > > > Dear all,
> > > >
> > > > After upgrading to ACS 4.8.1, one of our virtual router's /var/log
> > > > partition is always full and used up quite fast. This caused the VR
> not
> > > > able to serve DHCP and password requests from VM.
> > > >
> > > > root@r-4155-VM:/var/log# df -h
> > > > Filesystem Size Used
> > Avail
> > > > Use% Mounted on
> > > > rootfs 461M 158M
> > 280M
> > > > 37% /
> > > > udev 10M 0
> > > > 10M 0% /dev
> > > > tmpfs 25M 236K
> > > > 25M 1% /run
> > > > /dev/disk/by-uuid/30c81d3d-ee9f-4a88-81c1-5f349b22ba1d 461M 158M
> > 280M
> > > > 37% /
> > > > tmpfs 5.0M 0
> > > > 5.0M 0% /run/lock
> > > > tmpfs 157M 0
> > > > 157M 0% /run/shm
> > > > /dev/vda1 73M 23M
> > 47M
> > > > 33% /boot
> > > > /dev/vda6 92M 5.6M
> > > > 81M 7% /home
> > > > /dev/vda8 184M 6.2M
> > > > 169M 4% /opt
> > > > /dev/vda11 92M 5.6M
> > > > 81M 7% /tmp
> > > > /dev/vda7 751M 493M
> > 219M
> > > > 70% /usr
> > > > /dev/vda9 563M 282M
> > 252M
> > > > 53% /var
> > > > /dev/vda10 184M 176M
> > 0
> > > > 100% /var/log
> > > >
> > > > Even after rotating and clearing the logs, the usage of /var/log is
> > only
> > > > 4.7M so I am not too sure where is the 176M coming from.
> > > >
> > > > root@r-4155-VM:/var/log# du -h
> > > > 1.0K ./samba
> > > > 3.8M ./sysstat
> > > > 68K ./apt
> > > > 7.0K ./apache2
> > > > 3.0K ./fsck
> > > > 317K ./installer/cdebconf
> > > > 809K ./installer
> > > > 1.0K ./news
> > > > 12K ./lost+found
> > > > 1.0K ./ntpstats
> > > > 4.7M .
> > > >
> > > > /dev/vda10 184M 175M
> > 475K
> > > > 100% /var/log
> > > >
> > > > I would need to clear the logs and do a "service dnsmasq restart"
> > > > regularly to make the VR functioning again, which is quite
> troublesome.
> > > >
> > > > Any advice is greatly appreciated.
> > > >
> > > > Looking forward to your reply, thank you.
> > > >
> > > > Cheers.
> > > >
> > >
> >
>
Re: ACS 4.8.1: Virtual router - /var/log always full
Posted by Wei ZHOU <us...@gmail.com>.
This is because the free space in systemvm template is not big, and
logrotate rules are not wise enough to hold the logs.
you can remove the "delaycompress" and reduce the "size" in
systemvm/patches/debian/config/etc/logrotate.d/rsyslog
(/etc/logrotate.d/rsyslog in VR/systemvms), then restart logrotate to fix
it.
-Wei
2016-11-28 5:45 GMT+01:00 Will Stevens <ws...@cloudops.com>:
> Are you using XenServer? We recently found a bug in XenServer which has
> similar characteristics. I can put together details of the problem if
> yes...
>
> Cheers,
>
> Will
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Sun, Nov 27, 2016 at 11:36 PM, Cloud List <cl...@sg.or.id> wrote:
>
> > Dear all,
> >
> > Please ignore this, managed to find the reason - paswd_server_ip process
> is
> > holding to a log file which has been deleted earlier to fix the disk
> space
> > issue. Restarted the process and the disk space is fine now.
> >
> > ===
> > passwd_se 2199 root 1w REG 254,10
> > 171802407 15 /var/log/cloud.log (deleted)
> > passwd_se 2199 root 2w REG 254,10
> > 171802407 15 /var/log/cloud.log (deleted)
> > passwd_se 2199 root 3w REG 254,10
> > 171802407 15 /var/log/cloud.log (deleted)
> > python 2202 root 3w REG 254,10
> > 171802407 15 /var/log/cloud.log (deleted)
> >
> > 2199 ? S 0:00 /bin/bash /opt/cloud/bin/passwd_server_ip
> > X.X.X.2 dummy
> > 2202 ? S 7:13 python /opt/cloud/bin/passwd_server_ip.py
> > X.X.X.2
> > ===
> >
> > Thank you.
> >
> >
> > On Mon, Nov 28, 2016 at 11:46 AM, Cloud List <cl...@sg.or.id>
> wrote:
> >
> > > Dear all,
> > >
> > > After upgrading to ACS 4.8.1, one of our virtual router's /var/log
> > > partition is always full and used up quite fast. This caused the VR not
> > > able to serve DHCP and password requests from VM.
> > >
> > > root@r-4155-VM:/var/log# df -h
> > > Filesystem Size Used
> Avail
> > > Use% Mounted on
> > > rootfs 461M 158M
> 280M
> > > 37% /
> > > udev 10M 0
> > > 10M 0% /dev
> > > tmpfs 25M 236K
> > > 25M 1% /run
> > > /dev/disk/by-uuid/30c81d3d-ee9f-4a88-81c1-5f349b22ba1d 461M 158M
> 280M
> > > 37% /
> > > tmpfs 5.0M 0
> > > 5.0M 0% /run/lock
> > > tmpfs 157M 0
> > > 157M 0% /run/shm
> > > /dev/vda1 73M 23M
> 47M
> > > 33% /boot
> > > /dev/vda6 92M 5.6M
> > > 81M 7% /home
> > > /dev/vda8 184M 6.2M
> > > 169M 4% /opt
> > > /dev/vda11 92M 5.6M
> > > 81M 7% /tmp
> > > /dev/vda7 751M 493M
> 219M
> > > 70% /usr
> > > /dev/vda9 563M 282M
> 252M
> > > 53% /var
> > > /dev/vda10 184M 176M
> 0
> > > 100% /var/log
> > >
> > > Even after rotating and clearing the logs, the usage of /var/log is
> only
> > > 4.7M so I am not too sure where is the 176M coming from.
> > >
> > > root@r-4155-VM:/var/log# du -h
> > > 1.0K ./samba
> > > 3.8M ./sysstat
> > > 68K ./apt
> > > 7.0K ./apache2
> > > 3.0K ./fsck
> > > 317K ./installer/cdebconf
> > > 809K ./installer
> > > 1.0K ./news
> > > 12K ./lost+found
> > > 1.0K ./ntpstats
> > > 4.7M .
> > >
> > > /dev/vda10 184M 175M
> 475K
> > > 100% /var/log
> > >
> > > I would need to clear the logs and do a "service dnsmasq restart"
> > > regularly to make the VR functioning again, which is quite troublesome.
> > >
> > > Any advice is greatly appreciated.
> > >
> > > Looking forward to your reply, thank you.
> > >
> > > Cheers.
> > >
> >
>
Re: ACS 4.8.1: Virtual router - /var/log always full
Posted by Will Stevens <ws...@cloudops.com>.
Are you using XenServer? We recently found a bug in XenServer which has
similar characteristics. I can put together details of the problem if
yes...
Cheers,
Will
*Will STEVENS*
Lead Developer
<https://goo.gl/NYZ8KK>
On Sun, Nov 27, 2016 at 11:36 PM, Cloud List <cl...@sg.or.id> wrote:
> Dear all,
>
> Please ignore this, managed to find the reason - paswd_server_ip process is
> holding to a log file which has been deleted earlier to fix the disk space
> issue. Restarted the process and the disk space is fine now.
>
> ===
> passwd_se 2199 root 1w REG 254,10
> 171802407 15 /var/log/cloud.log (deleted)
> passwd_se 2199 root 2w REG 254,10
> 171802407 15 /var/log/cloud.log (deleted)
> passwd_se 2199 root 3w REG 254,10
> 171802407 15 /var/log/cloud.log (deleted)
> python 2202 root 3w REG 254,10
> 171802407 15 /var/log/cloud.log (deleted)
>
> 2199 ? S 0:00 /bin/bash /opt/cloud/bin/passwd_server_ip
> X.X.X.2 dummy
> 2202 ? S 7:13 python /opt/cloud/bin/passwd_server_ip.py
> X.X.X.2
> ===
>
> Thank you.
>
>
> On Mon, Nov 28, 2016 at 11:46 AM, Cloud List <cl...@sg.or.id> wrote:
>
> > Dear all,
> >
> > After upgrading to ACS 4.8.1, one of our virtual router's /var/log
> > partition is always full and used up quite fast. This caused the VR not
> > able to serve DHCP and password requests from VM.
> >
> > root@r-4155-VM:/var/log# df -h
> > Filesystem Size Used Avail
> > Use% Mounted on
> > rootfs 461M 158M 280M
> > 37% /
> > udev 10M 0
> > 10M 0% /dev
> > tmpfs 25M 236K
> > 25M 1% /run
> > /dev/disk/by-uuid/30c81d3d-ee9f-4a88-81c1-5f349b22ba1d 461M 158M 280M
> > 37% /
> > tmpfs 5.0M 0
> > 5.0M 0% /run/lock
> > tmpfs 157M 0
> > 157M 0% /run/shm
> > /dev/vda1 73M 23M 47M
> > 33% /boot
> > /dev/vda6 92M 5.6M
> > 81M 7% /home
> > /dev/vda8 184M 6.2M
> > 169M 4% /opt
> > /dev/vda11 92M 5.6M
> > 81M 7% /tmp
> > /dev/vda7 751M 493M 219M
> > 70% /usr
> > /dev/vda9 563M 282M 252M
> > 53% /var
> > /dev/vda10 184M 176M 0
> > 100% /var/log
> >
> > Even after rotating and clearing the logs, the usage of /var/log is only
> > 4.7M so I am not too sure where is the 176M coming from.
> >
> > root@r-4155-VM:/var/log# du -h
> > 1.0K ./samba
> > 3.8M ./sysstat
> > 68K ./apt
> > 7.0K ./apache2
> > 3.0K ./fsck
> > 317K ./installer/cdebconf
> > 809K ./installer
> > 1.0K ./news
> > 12K ./lost+found
> > 1.0K ./ntpstats
> > 4.7M .
> >
> > /dev/vda10 184M 175M 475K
> > 100% /var/log
> >
> > I would need to clear the logs and do a "service dnsmasq restart"
> > regularly to make the VR functioning again, which is quite troublesome.
> >
> > Any advice is greatly appreciated.
> >
> > Looking forward to your reply, thank you.
> >
> > Cheers.
> >
>
Re: ACS 4.8.1: Virtual router - /var/log always full
Posted by Will Stevens <ws...@cloudops.com>.
Are you using XenServer? We recently found a bug in XenServer which has
similar characteristics. I can put together details of the problem if
yes...
Cheers,
Will
*Will STEVENS*
Lead Developer
<https://goo.gl/NYZ8KK>
On Sun, Nov 27, 2016 at 11:36 PM, Cloud List <cl...@sg.or.id> wrote:
> Dear all,
>
> Please ignore this, managed to find the reason - paswd_server_ip process is
> holding to a log file which has been deleted earlier to fix the disk space
> issue. Restarted the process and the disk space is fine now.
>
> ===
> passwd_se 2199 root 1w REG 254,10
> 171802407 15 /var/log/cloud.log (deleted)
> passwd_se 2199 root 2w REG 254,10
> 171802407 15 /var/log/cloud.log (deleted)
> passwd_se 2199 root 3w REG 254,10
> 171802407 15 /var/log/cloud.log (deleted)
> python 2202 root 3w REG 254,10
> 171802407 15 /var/log/cloud.log (deleted)
>
> 2199 ? S 0:00 /bin/bash /opt/cloud/bin/passwd_server_ip
> X.X.X.2 dummy
> 2202 ? S 7:13 python /opt/cloud/bin/passwd_server_ip.py
> X.X.X.2
> ===
>
> Thank you.
>
>
> On Mon, Nov 28, 2016 at 11:46 AM, Cloud List <cl...@sg.or.id> wrote:
>
> > Dear all,
> >
> > After upgrading to ACS 4.8.1, one of our virtual router's /var/log
> > partition is always full and used up quite fast. This caused the VR not
> > able to serve DHCP and password requests from VM.
> >
> > root@r-4155-VM:/var/log# df -h
> > Filesystem Size Used Avail
> > Use% Mounted on
> > rootfs 461M 158M 280M
> > 37% /
> > udev 10M 0
> > 10M 0% /dev
> > tmpfs 25M 236K
> > 25M 1% /run
> > /dev/disk/by-uuid/30c81d3d-ee9f-4a88-81c1-5f349b22ba1d 461M 158M 280M
> > 37% /
> > tmpfs 5.0M 0
> > 5.0M 0% /run/lock
> > tmpfs 157M 0
> > 157M 0% /run/shm
> > /dev/vda1 73M 23M 47M
> > 33% /boot
> > /dev/vda6 92M 5.6M
> > 81M 7% /home
> > /dev/vda8 184M 6.2M
> > 169M 4% /opt
> > /dev/vda11 92M 5.6M
> > 81M 7% /tmp
> > /dev/vda7 751M 493M 219M
> > 70% /usr
> > /dev/vda9 563M 282M 252M
> > 53% /var
> > /dev/vda10 184M 176M 0
> > 100% /var/log
> >
> > Even after rotating and clearing the logs, the usage of /var/log is only
> > 4.7M so I am not too sure where is the 176M coming from.
> >
> > root@r-4155-VM:/var/log# du -h
> > 1.0K ./samba
> > 3.8M ./sysstat
> > 68K ./apt
> > 7.0K ./apache2
> > 3.0K ./fsck
> > 317K ./installer/cdebconf
> > 809K ./installer
> > 1.0K ./news
> > 12K ./lost+found
> > 1.0K ./ntpstats
> > 4.7M .
> >
> > /dev/vda10 184M 175M 475K
> > 100% /var/log
> >
> > I would need to clear the logs and do a "service dnsmasq restart"
> > regularly to make the VR functioning again, which is quite troublesome.
> >
> > Any advice is greatly appreciated.
> >
> > Looking forward to your reply, thank you.
> >
> > Cheers.
> >
>
Re: ACS 4.8.1: Virtual router - /var/log always full
Posted by Cloud List <cl...@sg.or.id>.
Dear all,
Please ignore this, managed to find the reason - paswd_server_ip process is
holding to a log file which has been deleted earlier to fix the disk space
issue. Restarted the process and the disk space is fine now.
===
passwd_se 2199 root 1w REG 254,10
171802407 15 /var/log/cloud.log (deleted)
passwd_se 2199 root 2w REG 254,10
171802407 15 /var/log/cloud.log (deleted)
passwd_se 2199 root 3w REG 254,10
171802407 15 /var/log/cloud.log (deleted)
python 2202 root 3w REG 254,10
171802407 15 /var/log/cloud.log (deleted)
2199 ? S 0:00 /bin/bash /opt/cloud/bin/passwd_server_ip
X.X.X.2 dummy
2202 ? S 7:13 python /opt/cloud/bin/passwd_server_ip.py X.X.X.2
===
Thank you.
On Mon, Nov 28, 2016 at 11:46 AM, Cloud List <cl...@sg.or.id> wrote:
> Dear all,
>
> After upgrading to ACS 4.8.1, one of our virtual router's /var/log
> partition is always full and used up quite fast. This caused the VR not
> able to serve DHCP and password requests from VM.
>
> root@r-4155-VM:/var/log# df -h
> Filesystem Size Used Avail
> Use% Mounted on
> rootfs 461M 158M 280M
> 37% /
> udev 10M 0
> 10M 0% /dev
> tmpfs 25M 236K
> 25M 1% /run
> /dev/disk/by-uuid/30c81d3d-ee9f-4a88-81c1-5f349b22ba1d 461M 158M 280M
> 37% /
> tmpfs 5.0M 0
> 5.0M 0% /run/lock
> tmpfs 157M 0
> 157M 0% /run/shm
> /dev/vda1 73M 23M 47M
> 33% /boot
> /dev/vda6 92M 5.6M
> 81M 7% /home
> /dev/vda8 184M 6.2M
> 169M 4% /opt
> /dev/vda11 92M 5.6M
> 81M 7% /tmp
> /dev/vda7 751M 493M 219M
> 70% /usr
> /dev/vda9 563M 282M 252M
> 53% /var
> /dev/vda10 184M 176M 0
> 100% /var/log
>
> Even after rotating and clearing the logs, the usage of /var/log is only
> 4.7M so I am not too sure where is the 176M coming from.
>
> root@r-4155-VM:/var/log# du -h
> 1.0K ./samba
> 3.8M ./sysstat
> 68K ./apt
> 7.0K ./apache2
> 3.0K ./fsck
> 317K ./installer/cdebconf
> 809K ./installer
> 1.0K ./news
> 12K ./lost+found
> 1.0K ./ntpstats
> 4.7M .
>
> /dev/vda10 184M 175M 475K
> 100% /var/log
>
> I would need to clear the logs and do a "service dnsmasq restart"
> regularly to make the VR functioning again, which is quite troublesome.
>
> Any advice is greatly appreciated.
>
> Looking forward to your reply, thank you.
>
> Cheers.
>
Re: ACS 4.8.1: Virtual router - /var/log always full
Posted by Cloud List <cl...@sg.or.id>.
Dear all,
Please ignore this, managed to find the reason - paswd_server_ip process is
holding to a log file which has been deleted earlier to fix the disk space
issue. Restarted the process and the disk space is fine now.
===
passwd_se 2199 root 1w REG 254,10
171802407 15 /var/log/cloud.log (deleted)
passwd_se 2199 root 2w REG 254,10
171802407 15 /var/log/cloud.log (deleted)
passwd_se 2199 root 3w REG 254,10
171802407 15 /var/log/cloud.log (deleted)
python 2202 root 3w REG 254,10
171802407 15 /var/log/cloud.log (deleted)
2199 ? S 0:00 /bin/bash /opt/cloud/bin/passwd_server_ip
X.X.X.2 dummy
2202 ? S 7:13 python /opt/cloud/bin/passwd_server_ip.py X.X.X.2
===
Thank you.
On Mon, Nov 28, 2016 at 11:46 AM, Cloud List <cl...@sg.or.id> wrote:
> Dear all,
>
> After upgrading to ACS 4.8.1, one of our virtual router's /var/log
> partition is always full and used up quite fast. This caused the VR not
> able to serve DHCP and password requests from VM.
>
> root@r-4155-VM:/var/log# df -h
> Filesystem Size Used Avail
> Use% Mounted on
> rootfs 461M 158M 280M
> 37% /
> udev 10M 0
> 10M 0% /dev
> tmpfs 25M 236K
> 25M 1% /run
> /dev/disk/by-uuid/30c81d3d-ee9f-4a88-81c1-5f349b22ba1d 461M 158M 280M
> 37% /
> tmpfs 5.0M 0
> 5.0M 0% /run/lock
> tmpfs 157M 0
> 157M 0% /run/shm
> /dev/vda1 73M 23M 47M
> 33% /boot
> /dev/vda6 92M 5.6M
> 81M 7% /home
> /dev/vda8 184M 6.2M
> 169M 4% /opt
> /dev/vda11 92M 5.6M
> 81M 7% /tmp
> /dev/vda7 751M 493M 219M
> 70% /usr
> /dev/vda9 563M 282M 252M
> 53% /var
> /dev/vda10 184M 176M 0
> 100% /var/log
>
> Even after rotating and clearing the logs, the usage of /var/log is only
> 4.7M so I am not too sure where is the 176M coming from.
>
> root@r-4155-VM:/var/log# du -h
> 1.0K ./samba
> 3.8M ./sysstat
> 68K ./apt
> 7.0K ./apache2
> 3.0K ./fsck
> 317K ./installer/cdebconf
> 809K ./installer
> 1.0K ./news
> 12K ./lost+found
> 1.0K ./ntpstats
> 4.7M .
>
> /dev/vda10 184M 175M 475K
> 100% /var/log
>
> I would need to clear the logs and do a "service dnsmasq restart"
> regularly to make the VR functioning again, which is quite troublesome.
>
> Any advice is greatly appreciated.
>
> Looking forward to your reply, thank you.
>
> Cheers.
>