You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by serbaut <gi...@git.apache.org> on 2017/01/23 13:22:59 UTC
[GitHub] cloudstack pull request #1915: CLOUDSTACK-9746 system-vm: logrotate config c...
GitHub user serbaut opened a pull request:
https://github.com/apache/cloudstack/pull/1915
CLOUDSTACK-9746 system-vm: logrotate config causes critical failures
* rotate both daily and by size by using maxsize in stead of size
* decrease the max size to 10M and reduce the number of kept files to a maximum of 5
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/serbaut/cloudstack CLOUDSTACK-9746
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/cloudstack/pull/1915.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1915
----
commit de9c0ab0a4cf31f097c9f7e7039add4240a62ac9
Author: Joakim Sernbrant <jo...@trioptima.com>
Date: 2017-01-23T11:13:15Z
CLOUDSTACK-9746 system-vm: logrotate config causes critical failures
* rotate both daily and by size by using maxsize in stead of size
* decrease the max size to 10M and reduce the number of kept files to a maximum of 5
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...
Posted by ustcweizhou <gi...@git.apache.org>.
Github user ustcweizhou commented on the issue:
https://github.com/apache/cloudstack/pull/1915
@dmabry I will test it and update you.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...
Posted by ustcweizhou <gi...@git.apache.org>.
Github user ustcweizhou commented on the issue:
https://github.com/apache/cloudstack/pull/1915
@serbaut I agree with reducing the size.
However, I suggest to
(1) remove the delaycompress, then the files like syslog.1 or rsyslog.1 will be compressed.
(2) keep rotate , as the compressed file are quite small (< 1MB). Keeping more compressing files (especially cloud.log) will be helpful for troubleshooting in case we have some issues.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...
Posted by serbaut <gi...@git.apache.org>.
Github user serbaut commented on the issue:
https://github.com/apache/cloudstack/pull/1915
Ok. I removed it from rsyslog since it should be safe there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request #1915: CLOUDSTACK-9746 system-vm: logrotate config c...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/cloudstack/pull/1915
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...
Posted by ustcweizhou <gi...@git.apache.org>.
Github user ustcweizhou commented on the issue:
https://github.com/apache/cloudstack/pull/1915
agree with @dmabry
logrotate need to be moved (or copied) to cron.hourly
```
diff --git a/systemvm/patches/debian/config/etc/cron.hourly/logrotate b/systemvm/patches/debian/config/etc/cron.hourly/logrotate
new file mode 100755
index 0000000..055b4d5
--- /dev/null
+++ b/systemvm/patches/debian/config/etc/cron.hourly/logrotate
@@ -0,0 +1,5 @@
+#!/bin/sh
+
+test -x /usr/sbin/logrotate || exit 0
+/usr/sbin/logrotate /etc/logrotate.conf
+
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...
Posted by dmabry <gi...@git.apache.org>.
Github user dmabry commented on the issue:
https://github.com/apache/cloudstack/pull/1915
@ustcweizhou Thanks for taking the time to test as well. I confirm in our testing over the weekend that we were able to consistently create/destroy/expunge VMs every 2 minutes without issue. Before the PR https://github.com/apache/cloudstack/pull/1954 and this PR, we were having the VPC VR fall over ~8 hours into the test.
In summary, this PR with the one I submitted this morning solved our issues.
Thanks again for the help.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...
Posted by serbaut <gi...@git.apache.org>.
Github user serbaut commented on the issue:
https://github.com/apache/cloudstack/pull/1915
Is it safe to remove delaycompress across the board, I assume it is there for a reason?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...
Posted by dmabry <gi...@git.apache.org>.
Github user dmabry commented on the issue:
https://github.com/apache/cloudstack/pull/1915
I have submitted a PR against master that fixes cloud-early-config.
https://github.com/apache/cloudstack/pull/1954
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...
Posted by dmabry <gi...@git.apache.org>.
Github user dmabry commented on the issue:
https://github.com/apache/cloudstack/pull/1915
@serbaut Just curious, are you using VPC routers?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...
Posted by dmabry <gi...@git.apache.org>.
Github user dmabry commented on the issue:
https://github.com/apache/cloudstack/pull/1915
LGTM. tag:mergeready
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...
Posted by serbaut <gi...@git.apache.org>.
Github user serbaut commented on the issue:
https://github.com/apache/cloudstack/pull/1915
@dmabry yes we are using redundant VPC routers.
Using `size` instead of `maxsize` would delay the rotation of logs until they reach `size`. 10 logfiles with size 10MB could consume 200MB due to deplaycompress. Using `maxsize` would rotate the slowly growing logfiles daily and avoid this issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...
Posted by ustcweizhou <gi...@git.apache.org>.
Github user ustcweizhou commented on the issue:
https://github.com/apache/cloudstack/pull/1915
@serbaut maxsize is fine for me.
Could you please remove delaycompress, and increase rotate in systemvm/patches/debian/config/etc/logrotate.d/cloud (eg from 4 to 10) ?
LTGM after that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...
Posted by leprechau <gi...@git.apache.org>.
Github user leprechau commented on the issue:
https://github.com/apache/cloudstack/pull/1915
We always want `compress` ... but the only time you need or want `delaycompress` is if you can't be sure that the program writing to the log can be successfully told to stop appending to that log. In the case where you are certain that the writing program is going to do the right thing there is no need to add `delaycompress` as it just takes up extra space in already rotated logs until the next iteration.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...
Posted by kiwiflyer <gi...@git.apache.org>.
Github user kiwiflyer commented on the issue:
https://github.com/apache/cloudstack/pull/1915
We pulled this in and tested it alongside https://github.com/apache/cloudstack/pull/1954
We did remove the delay compress in our testing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...
Posted by dmabry <gi...@git.apache.org>.
Github user dmabry commented on the issue:
https://github.com/apache/cloudstack/pull/1915
@serbaut Can you do a force push to kick off jenkins again. I'm guessing Jenkins just had an issue and not the PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...
Posted by dmabry <gi...@git.apache.org>.
Github user dmabry commented on the issue:
https://github.com/apache/cloudstack/pull/1915
@ustcweizhou - Thanks for the +1 on cron.hourly move. From looking at the code, it looks like non-redundant VRs are already moving to cron.hourly, but vpcrouter isn't. In our case, we are using vpcrouter. I'm working on a PR against master that will fix the vpcrouter flavor of this code. I'll link the PR here when I'm done testing.
I'd also like to understand why you would prefer 'size' over 'maxsize'. I know the difference is subtle and it really won't apply to anything that is set to "hourly", but it could impact the dailys. In the VR case with a limited /var/log partition size, I would think that the 'maxsize' option would be preferable. It would keep run away log files in check by rotating them even before their interval is reached.
```size size
Log files are rotated only if they grow bigger then size bytes.
If size is followed by k, the size is assumed to be in kilo-
bytes. If the M is used, the size is in megabytes, and if G is
used, the size is in gigabytes. So size 100, size 100k, size
100M and size 100G are all valid.
maxsize size
Log files are rotated when they grow bigger than size bytes even before
the additionally specified time interval (daily, weekly, monthly,
or yearly). The related size option is similar except that it
is mutually exclusive with the time interval options, and it causes
log files to be rotated without regard for the last rotation time.
When maxsize is used, both the size and timestamp of a log file are
considered.
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...
Posted by dmabry <gi...@git.apache.org>.
Github user dmabry commented on the issue:
https://github.com/apache/cloudstack/pull/1915
@serbaut I agree with @ustcweizhou. Please remove delaycompress and up to 10. I'd like to get this PR in as it is the second part of the problem resolution for my issue. After that LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...
Posted by rhtyd <gi...@git.apache.org>.
Github user rhtyd commented on the issue:
https://github.com/apache/cloudstack/pull/1915
LGTM @karuturi this is ready for merge
Ping @abhinandanprateek in case you want to review?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...
Posted by dmabry <gi...@git.apache.org>.
Github user dmabry commented on the issue:
https://github.com/apache/cloudstack/pull/1915
Agreed with @leprechau . I don't think just changing logrotate to compress and rotate at smaller size intervals will not completely solve our issues here. We really need to adjust logrotate to run every hour, or make cloud.log less verbose by default. I'm looking into seeing if it would be viable to make logrotate run every hour, but I believe that will require changing how the systemvm image is built by moving logrotate from cron.daily to cron.hourly. Not a huge deal, but I think it might be right approach over all since we have limited disk space in /var/log.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...
Posted by ustcweizhou <gi...@git.apache.org>.
Github user ustcweizhou commented on the issue:
https://github.com/apache/cloudstack/pull/1915
1. As @dmabry said, the logrotate is moved from cron.daily to cron.hourly in isolated network VR in commit.
We should make similar change for VPC VR and VR for shared network.
```
commit 6ac06e5e5e3ceed4a3e3a86ea5f82ffb59c266f2
Author: Abhinandan Prateek <ab...@shapeblue.com>
AuthorDate: Mon Mar 30 11:36:11 2015 +0530
Commit: Abhinandan Prateek <ab...@shapeblue.com>
CommitDate: Mon Mar 30 13:49:23 2015 +0530
CLOUDSTACK-6885: rotate logs hourly on VR
---
.../debian/config/etc/init.d/cloud-early-config | 4 ++++
systemvm/patches/debian/config/etc/logrotate.conf | 2 +-
.../patches/debian/config/etc/logrotate.d/cloud | 3 ---
.../patches/debian/config/etc/logrotate.d/rsyslog | 2 ++
4 files changed, 7 insertions(+), 4 deletions(-)
diff --git a/systemvm/patches/debian/config/etc/init.d/cloud-early-config b/systemvm/patches/debian/config/etc/init.d/cloud-early-config
index 729b58f..5a2ee6f 100755
--- a/systemvm/patches/debian/config/etc/init.d/cloud-early-config
+++ b/systemvm/patches/debian/config/etc/init.d/cloud-early-config
@@ -995,6 +995,10 @@ setup_router() {
sed -i "s/-A INPUT -i eth0 -p udp -m udp --dport 53 -j ACCEPT/-A INPUT -i eth0 -p udp -m udp --dport 53 -s $DHCP_RANGE\/$CIDR_SIZE -j ACCEPT/g" /etc/iptables/rules
sed -i "s/-A INPUT -i eth0 -p tcp -m tcp --dport 53 -j ACCEPT/-A INPUT -i eth0 -p tcp -m tcp --dport 53 -s $DHCP_RANGE\/$CIDR_SIZE -j ACCEPT/g" /etc/iptables/rules.v4
sed -i "s/-A INPUT -i eth0 -p tcp -m tcp --dport 53 -j ACCEPT/-A INPUT -i eth0 -p tcp -m tcp --dport 53 -s $DHCP_RANGE\/$CIDR_SIZE -j ACCEPT/g" /etc/iptables/rules
+
+ #setup hourly logrotate
+ mv -n /etc/cron.daily/logrotate /etc/cron.hourly 2>&1
+
}
```
2. size and maxsize have no difference in my testing. Anyway, maxsize is ok for me.
3. As I said in previous comment, we'd better use more 'rotate' for cloud.log
4. @dmabry If you are not confident with hourly file compress, we can add a file in /etc/cron.d/
```
# cat /etc/cron.d/logrorate.cloud
* * * * * root /usr/sbin/logrotate /etc/logrotate.d/cloud 2>&1 > /dev/null
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1915: CLOUDSTACK-9746 system-vm: logrotate config causes c...
Posted by leprechau <gi...@git.apache.org>.
Github user leprechau commented on the issue:
https://github.com/apache/cloudstack/pull/1915
We are seeing issues under heavy user interaction cycles with redundant VRs. In testing you can reliably completely fill the `/var/log/` partition with data primarily from `cloud.log` when you create/stop/delete/expunge ~200 VMs over a 24 hour period with the default settings.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---