You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by remibergsma <gi...@git.apache.org> on 2015/12/13 12:01:51 UTC
[GitHub] cloudstack pull request: CLOUDSTACK-9155 make sure logrotate is ef...
GitHub user remibergsma opened a pull request:
https://github.com/apache/cloudstack/pull/1235
CLOUDSTACK-9155 make sure logrotate is effective for cloud.log
Many processes on the VRs log to cloud.log. When log rotate kicks in, the file is rotated but the scripts still write to the old inode (cloud.log.1 after rotate). Tis quickly fills up the tiny log partition.
Using 'copytruncate' is a small tradeoff, there is a slight change of missing a log entry, but in the old situation nothing ended up in cloud.log after rotate (except for stuff that was (re)started) so I think this is the best solution until we properly rewrite the script to either use their own script or syslog.
More details: https://issues.apache.org/jira/browse/CLOUDSTACK-9155
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/remibergsma/cloudstack fix-passwd-server-logging
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/cloudstack/pull/1235.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1235
----
commit 0742535e13c51372c63b045181d6a20198fb397e
Author: Remi Bergsma <gi...@remi.nl>
Date: 2015-12-13T10:53:45Z
CLOUDSTACK-9155 make sure logrotate is effective
Many processes on the VRs log to cloud.log. When logrotate
kicks in, the file is rotated but the scripts still write
to the old inode (cloud.log.1 after rotate). Tis quickly
fills up the tiny log partition.
Using 'copytruncate' is a tradeoff, there is a slight
change of missing a log entry, but in the old situation
we were missing all of them after logrotate.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9155 make sure logrotate is ef...
Posted by borisroman <gi...@git.apache.org>.
Github user borisroman commented on the pull request:
https://github.com/apache/cloudstack/pull/1235#issuecomment-164255956
Based on explanation and code review: LGTM :+1:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9155 make sure logrotate is ef...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/cloudstack/pull/1235
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9155 make sure logrotate is ef...
Posted by wilderrodrigues <gi...@git.apache.org>.
Github user wilderrodrigues commented on the pull request:
https://github.com/apache/cloudstack/pull/1235#issuecomment-164253492
The change LGTM. We should get this in and for ACS 4.8 we do a refactor in order to have a proper solution in place.
Cheers,
Wilder
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9155 make sure logrotate is ef...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on the pull request:
https://github.com/apache/cloudstack/pull/1235#issuecomment-164251561
@remibergsma sounds like the scripts that are the problem do an open for write instead of log-call, do they? I saw your list of processes and most of them seem legit. Any idea which is the culprit. It seems like a quickfix.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9155 make sure logrotate is ef...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on the pull request:
https://github.com/apache/cloudstack/pull/1235#issuecomment-164253411
@remibergsma agree and let's not close the jira ticket on merging as this obviously needs tlc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9155 make sure logrotate is ef...
Posted by remibergsma <gi...@git.apache.org>.
Github user remibergsma commented on the pull request:
https://github.com/apache/cloudstack/pull/1235#issuecomment-164258629
Integration tests make no real sense, but tried a few to be sure:
```
Check the password file in the Router VM ... === TestName: test_isolate_network_password_server | Status : SUCCESS ===
ok
Create a redundant VPC with two networks with two VMs in each network ... === TestName: test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | Status : SUCCESS ===
ok
Create a redundant VPC with two networks with two VMs in each network and check default routes ... === TestName: test_02_redundant_VPC_default_routes | Status : SUCCESS ===
ok
Create a redundant VPC with two networks with two VMs in each network ... === TestName: test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers | Status : SUCCESS ===
ok
Test iptables default INPUT/FORWARD policy on RouterVM ... === TestName: test_02_routervm_iptables_policies | Status : SUCCESS ===
```
Looking good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9155 make sure logrotate is ef...
Posted by remibergsma <gi...@git.apache.org>.
Github user remibergsma commented on the pull request:
https://github.com/apache/cloudstack/pull/1235#issuecomment-164253007
@DaanHoogland There are many different solutions in place, some are bash/python scripts that do `>/var/log/cloud.log` etc, some use `nohup` to keep running and others are daemons that also log into the same file. There is no easy way to make logrotate restart those without some refactoring.
Also, close to release I didn't want to refactor the whole thing, as we'll have regression issues from that if we do it too quickly. This seems a safe fix for now, we should refactor it in the next release. Using syslog could be a simple solution, as that handles many of these scenarios already.
This is the best way to resolve the issue now IMHO.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9155 make sure logrotate is ef...
Posted by remibergsma <gi...@git.apache.org>.
Github user remibergsma commented on the pull request:
https://github.com/apache/cloudstack/pull/1235#issuecomment-164255438
The new script ends up on the router:
```
root@r-15-VM:~# cat /etc/logrotate.d/cloud
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
/var/log/cloud.log {
rotate 4
daily
size 10M
missingok
notifempty
compress
delaycompress
# CLOUDSTACK-9155: We cannot tell the processes that are writing to this
# file to use the new inode, so instead we copy the original file, truncate
# it and keep the same inode.
copytruncate
}
```
Truncated list of processes that have `cloud.log` open:
```
apache2 3636 3686 www-data 3w REG 254,10 264891 59 /var/log/cloud.log
apache2 3636 3687 www-data 3w REG 254,10 265499 59 /var/log/cloud.log
apache2 3636 3688 www-data 3w REG 254,10 265499 59 /var/log/cloud.log
apache2 3636 3689 www-data 3w REG 254,10 265499 59 /var/log/cloud.log
apache2 3636 3690 www-data 3w REG 254,10 265499 59 /var/log/cloud.log
conntrack 3837 root 3w REG 254,10 265499 59 /var/log/cloud.log
keepalive 3851 root 3w REG 254,10 265499 59 /var/log/cloud.log
keepalive 3852 root 3w REG 254,10 265499 59 /var/log/cloud.log
keepalive 3853 root 3w REG 254,10 265499 59 /var/log/cloud.log
_plutorun 4141 root 3w REG 254,10 265499 59 /var/log/cloud.log
logger 4142 root 3w REG 254,10 265499 59 /var/log/cloud.log
_plutorun 4145 root 3w REG 254,10 265499 59 /var/log/cloud.log
_plutoloa 4146 root 3w REG 254,10 265499 59 /var/log/cloud.log
xl2tpd 4203 root 3w REG 254,10 265499 59 /var/log/cloud.log
passwd_se 4225 root 1w REG 254,10 265499 59 /var/log/cloud.log
passwd_se 4225 root 2w REG 254,10 265499 59 /var/log/cloud.log
passwd_se 4225 root 3w REG 254,10 265499 59 /var/log/cloud.log
python 4229 root 3w REG 254,10 265499 59 /var/log/cloud.log
update_co 4927 root 3r REG 254,10 265499 59 /var/log/cloud.log
```
Force logrotate:
```
logrotate /etc/logrotate.conf -f
```
Same processes, they still have `cloud.log` open (instead of `cloud.log.1`)
```
apache2 3636 3686 www-data 3w REG 254,10 264891 59 /var/log/cloud.log
apache2 3636 3687 www-data 3w REG 254,10 265499 59 /var/log/cloud.log
apache2 3636 3688 www-data 3w REG 254,10 265499 59 /var/log/cloud.log
apache2 3636 3689 www-data 3w REG 254,10 265499 59 /var/log/cloud.log
apache2 3636 3690 www-data 3w REG 254,10 265499 59 /var/log/cloud.log
conntrack 3837 root 3w REG 254,10 265499 59 /var/log/cloud.log
keepalive 3851 root 3w REG 254,10 265499 59 /var/log/cloud.log
keepalive 3852 root 3w REG 254,10 265499 59 /var/log/cloud.log
keepalive 3853 root 3w REG 254,10 265499 59 /var/log/cloud.log
_plutorun 4141 root 3w REG 254,10 265499 59 /var/log/cloud.log
logger 4142 root 3w REG 254,10 265499 59 /var/log/cloud.log
_plutorun 4145 root 3w REG 254,10 265499 59 /var/log/cloud.log
_plutoloa 4146 root 3w REG 254,10 265499 59 /var/log/cloud.log
xl2tpd 4203 root 3w REG 254,10 265499 59 /var/log/cloud.log
passwd_se 4225 root 1w REG 254,10 265499 59 /var/log/cloud.log
passwd_se 4225 root 2w REG 254,10 265499 59 /var/log/cloud.log
passwd_se 4225 root 3w REG 254,10 265499 59 /var/log/cloud.log
python 4229 root 3w REG 254,10 265499 59 /var/log/cloud.log
update_co 4927 root 3r REG 254,10 265499 59 /var/log/cloud.log
```
Deploying this to our clouds running 4.7.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9155 make sure logrotate is ef...
Posted by remibergsma <gi...@git.apache.org>.
Github user remibergsma commented on the pull request:
https://github.com/apache/cloudstack/pull/1235#issuecomment-164254871
Before PR this was seen after log rotate:
```
root@r-996-VM:/etc# lsof | grep cloud.log.1
_plutorun 767 root 3w REG 202,10 26054919 71 /var/log/cloud.log.1
logger 768 root 3w REG 202,10 26054919 71 /var/log/cloud.log.1
_plutorun 772 root 3w REG 202,10 26054919 71 /var/log/cloud.log.1
_plutoloa 773 root 3w REG 202,10 26054919 71 /var/log/cloud.log.1
xl2tpd 843 root 3w REG 202,10 26054919 71 /var/log/cloud.log.1
python 854 root 3w REG 202,10 26054919 71 /var/log/cloud.log.1
passwd_se 860 root 1w REG 202,10 26054919 71 /var/log/cloud.log.1
passwd_se 860 root 2w REG 202,10 26054919 71 /var/log/cloud.log.1
passwd_se 860 root 3w REG 202,10 26054919 71 /var/log/cloud.log.1
python 863 root 3w REG 202,10 26054919 71 /var/log/cloud.log.1
passwd_se 869 root 1w REG 202,10 26054919 71 /var/log/cloud.log.1
passwd_se 869 root 2w REG 202,10 26054919 71 /var/log/cloud.log.1
passwd_se 869 root 3w REG 202,10 26054919 71 /var/log/cloud.log.1
python 871 root 3w REG 202,10 26054919 71 /var/log/cloud.log.1
```
After change:
```
root@r-996-VM:/etc# lsof | grep cloud.log.1
(no results)
```
They all still have `cloud.log` open.
Logrotate works:
```
root@r-996-VM:/var/log# ls -lah cloud*
-rw-r--r-- 1 root root 3.6M Dec 13 10:49 cloud_demo
-rw-r--r-- 1 root root 0 Dec 13 10:50 cloud.log
-rw-r--r-- 1 root root 18M Dec 13 10:50 cloud.log.1
-rw-r----- 1 root adm 0 Dec 13 10:16 cloud-service.log
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9155 make sure logrotate is ef...
Posted by remibergsma <gi...@git.apache.org>.
Github user remibergsma commented on the pull request:
https://github.com/apache/cloudstack/pull/1235#issuecomment-164247329
Pinging @wilderrodrigues @fborn @DaanHoogland @borisroman to review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9155 make sure logrotate is ef...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on the pull request:
https://github.com/apache/cloudstack/pull/1235#issuecomment-164255579
As you asked on im @remibergsma : LGTM more explicitely
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---