You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Joris van Lieshout <jv...@schubergphilis.com> on 2014/05/22 13:42:31 UTC
Review Request 21806: Disable conntrackd-stats logging to prevent it from
filling up /var
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21806/
-----------------------------------------------------------
Review request for cloudstack, Abhinandan Prateek, daan Hoogland, Rajesh Battala, Rohit Yadav, and Hugo Trippaers.
Repository: cloudstack-git
Description
-------
Conntrackd package has a bug where the comment in the default config file states that stats logging is disabled by default but the config parameter is set to on. The consequence for ACS is that a conntrackd-stats.log file is created during the build of the svm. This logfile gets rotated by logrotate which has a post action to restart conntrackd. Even if the svm is not a redundant router. On vpc routers for instance the stats logging file can grow quickly and fill up the /var volume killing the vm.
Diffs
-----
tools/appliance/definitions/systemvm64template/postinstall.sh cc8ead9
tools/appliance/definitions/systemvmtemplate/postinstall.sh 23e66dd
Diff: https://reviews.apache.org/r/21806/diff/
Testing
-------
We are running this fix in our production environment for a couple months allready.
Thanks,
Joris van Lieshout
Re: Review Request 21806: CLOUDSTACK-6751: Disable conntrackd-stats logging
to prevent it from filling up /var
Posted by ASF Subversion and Git Services <as...@urd.zones.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21806/#review43720
-----------------------------------------------------------
Commit 2c810d73e24411e767296b62e59d2acbb77b7df5 in cloudstack's branch refs/heads/master from Marcus Sorensen
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=2c810d7 ]
CLOUDSTACK-6751 - Disable stats logging for conntrackd upon systemvm creation
Submitted-by: Joris van Lieshout <jv...@schubergphilis.com>
- ASF Subversion and Git Services
On May 22, 2014, 2:35 p.m., Joris van Lieshout wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21806/
> -----------------------------------------------------------
>
> (Updated May 22, 2014, 2:35 p.m.)
>
>
> Review request for cloudstack, Abhinandan Prateek, daan Hoogland, Rajesh Battala, Rohit Yadav, and Hugo Trippaers.
>
>
> Bugs: CLOUDSTACK-6751
> https://issues.apache.org/jira/browse/CLOUDSTACK-6751
>
>
> Repository: cloudstack-git
>
>
> Description
> -------
>
> Conntrackd package has a bug where the comment in the default config file states that stats logging is disabled by default but the config parameter is set to on. The consequence for ACS is that a conntrackd-stats.log file is created during the build of the svm. This logfile gets rotated by logrotate which has a post action to restart conntrackd. Even if the svm is not a redundant router. On vpc routers for instance the stats logging file can grow quickly and fill up the /var volume killing the vm.
>
>
> Diffs
> -----
>
> tools/appliance/definitions/systemvm64template/postinstall.sh cc8ead9
> tools/appliance/definitions/systemvmtemplate/postinstall.sh 23e66dd
>
> Diff: https://reviews.apache.org/r/21806/diff/
>
>
> Testing
> -------
>
> We are running this fix in our production environment for a couple months allready.
>
>
> Thanks,
>
> Joris van Lieshout
>
>
Re: Review Request 21806: CLOUDSTACK-6751: Disable conntrackd-stats logging
to prevent it from filling up /var
Posted by Marcus Sorensen <sh...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21806/#review43721
-----------------------------------------------------------
Ship it!
Ship It!
- Marcus Sorensen
On May 22, 2014, 2:35 p.m., Joris van Lieshout wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21806/
> -----------------------------------------------------------
>
> (Updated May 22, 2014, 2:35 p.m.)
>
>
> Review request for cloudstack, Abhinandan Prateek, daan Hoogland, Rajesh Battala, Rohit Yadav, and Hugo Trippaers.
>
>
> Bugs: CLOUDSTACK-6751
> https://issues.apache.org/jira/browse/CLOUDSTACK-6751
>
>
> Repository: cloudstack-git
>
>
> Description
> -------
>
> Conntrackd package has a bug where the comment in the default config file states that stats logging is disabled by default but the config parameter is set to on. The consequence for ACS is that a conntrackd-stats.log file is created during the build of the svm. This logfile gets rotated by logrotate which has a post action to restart conntrackd. Even if the svm is not a redundant router. On vpc routers for instance the stats logging file can grow quickly and fill up the /var volume killing the vm.
>
>
> Diffs
> -----
>
> tools/appliance/definitions/systemvm64template/postinstall.sh cc8ead9
> tools/appliance/definitions/systemvmtemplate/postinstall.sh 23e66dd
>
> Diff: https://reviews.apache.org/r/21806/diff/
>
>
> Testing
> -------
>
> We are running this fix in our production environment for a couple months allready.
>
>
> Thanks,
>
> Joris van Lieshout
>
>
Re: Review Request 21806: CLOUDSTACK-6751: Disable conntrackd-stats logging
to prevent it from filling up /var
Posted by Joris van Lieshout <jv...@schubergphilis.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21806/
-----------------------------------------------------------
(Updated May 22, 2014, 2:35 p.m.)
Review request for cloudstack, Abhinandan Prateek, daan Hoogland, Rajesh Battala, Rohit Yadav, and Hugo Trippaers.
Summary (updated)
-----------------
CLOUDSTACK-6751: Disable conntrackd-stats logging to prevent it from filling up /var
Bugs: CLOUDSTACK-6751
https://issues.apache.org/jira/browse/CLOUDSTACK-6751
Repository: cloudstack-git
Description
-------
Conntrackd package has a bug where the comment in the default config file states that stats logging is disabled by default but the config parameter is set to on. The consequence for ACS is that a conntrackd-stats.log file is created during the build of the svm. This logfile gets rotated by logrotate which has a post action to restart conntrackd. Even if the svm is not a redundant router. On vpc routers for instance the stats logging file can grow quickly and fill up the /var volume killing the vm.
Diffs
-----
tools/appliance/definitions/systemvm64template/postinstall.sh cc8ead9
tools/appliance/definitions/systemvmtemplate/postinstall.sh 23e66dd
Diff: https://reviews.apache.org/r/21806/diff/
Testing
-------
We are running this fix in our production environment for a couple months allready.
Thanks,
Joris van Lieshout
Re: Review Request 21806: Disable conntrackd-stats logging to prevent it
from filling up /var
Posted by Joris van Lieshout <jv...@schubergphilis.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21806/
-----------------------------------------------------------
(Updated May 22, 2014, 2:34 p.m.)
Review request for cloudstack, Abhinandan Prateek, daan Hoogland, Rajesh Battala, Rohit Yadav, and Hugo Trippaers.
Changes
-------
create jira bug CLOUDSTACK-6751
Bugs: CLOUDSTACK-6751
https://issues.apache.org/jira/browse/CLOUDSTACK-6751
Repository: cloudstack-git
Description
-------
Conntrackd package has a bug where the comment in the default config file states that stats logging is disabled by default but the config parameter is set to on. The consequence for ACS is that a conntrackd-stats.log file is created during the build of the svm. This logfile gets rotated by logrotate which has a post action to restart conntrackd. Even if the svm is not a redundant router. On vpc routers for instance the stats logging file can grow quickly and fill up the /var volume killing the vm.
Diffs
-----
tools/appliance/definitions/systemvm64template/postinstall.sh cc8ead9
tools/appliance/definitions/systemvmtemplate/postinstall.sh 23e66dd
Diff: https://reviews.apache.org/r/21806/diff/
Testing
-------
We are running this fix in our production environment for a couple months allready.
Thanks,
Joris van Lieshout
Re: Review Request 21806: Disable conntrackd-stats logging to prevent it
from filling up /var
Posted by Joris van Lieshout <jv...@schubergphilis.com>.
- Joris
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21806/#review43710
-----------------------------------------------------------
On May 22, 2014, 2:34 p.m., Joris van Lieshout wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21806/
> -----------------------------------------------------------
>
> (Updated May 22, 2014, 2:34 p.m.)
>
>
> Review request for cloudstack, Abhinandan Prateek, daan Hoogland, Rajesh Battala, Rohit Yadav, and Hugo Trippaers.
>
>
> Bugs: CLOUDSTACK-6751
> https://issues.apache.org/jira/browse/CLOUDSTACK-6751
>
>
> Repository: cloudstack-git
>
>
> Description
> -------
>
> Conntrackd package has a bug where the comment in the default config file states that stats logging is disabled by default but the config parameter is set to on. The consequence for ACS is that a conntrackd-stats.log file is created during the build of the svm. This logfile gets rotated by logrotate which has a post action to restart conntrackd. Even if the svm is not a redundant router. On vpc routers for instance the stats logging file can grow quickly and fill up the /var volume killing the vm.
>
>
> Diffs
> -----
>
> tools/appliance/definitions/systemvm64template/postinstall.sh cc8ead9
> tools/appliance/definitions/systemvmtemplate/postinstall.sh 23e66dd
>
> Diff: https://reviews.apache.org/r/21806/diff/
>
>
> Testing
> -------
>
> We are running this fix in our production environment for a couple months allready.
>
>
> Thanks,
>
> Joris van Lieshout
>
>
Re: Review Request 21806: Disable conntrackd-stats logging to prevent it
from filling up /var
Posted by Joris van Lieshout <jv...@schubergphilis.com>.
> On May 22, 2014, 1:27 p.m., Marcus Sorensen wrote:
> > I thought I had fixed this awhile back, but it was from the other direction. Conntrackd no longer restarts on log rotate unless conntrackd was already running (6b7f91d7). I suppose this wouldn't be bad to put in as well, so long as we know that conntrackd will in fact get started when we want it (redundant virtual router). Another issue contributing to it was that we had split up the tiny disk into many partitions, leaving only a few hundred MB available and stranded across many parts of the file tree, since there are times when we want conntrackd we should probably have a system vm template that can handle its logfiles.
> >
> > You need to create a bug report that we can tie this to.
Thank you for your response. I see your commit on the logrotate script. Although that alone would fix it I agree with you that fixing it both ways is more reliable. :)
I will create a but report as requested.
Regarding your comment on disk space we had issues with that as wel. Not so much for /var but in fact /usr (review 21696 CLOUDSTACK-6716). In my opinion having a lean and mean svm is fine. A better solution, when it comes to logging, would be to send it to the control ip with syslog. We're working on a patch that gives you the opportunity to do without losing local logging. I'll include you in the review request one i've finished it.
- Joris
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21806/#review43710
-----------------------------------------------------------
On May 22, 2014, 2:34 p.m., Joris van Lieshout wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21806/
> -----------------------------------------------------------
>
> (Updated May 22, 2014, 2:34 p.m.)
>
>
> Review request for cloudstack, Abhinandan Prateek, daan Hoogland, Rajesh Battala, Rohit Yadav, and Hugo Trippaers.
>
>
> Bugs: CLOUDSTACK-6751
> https://issues.apache.org/jira/browse/CLOUDSTACK-6751
>
>
> Repository: cloudstack-git
>
>
> Description
> -------
>
> Conntrackd package has a bug where the comment in the default config file states that stats logging is disabled by default but the config parameter is set to on. The consequence for ACS is that a conntrackd-stats.log file is created during the build of the svm. This logfile gets rotated by logrotate which has a post action to restart conntrackd. Even if the svm is not a redundant router. On vpc routers for instance the stats logging file can grow quickly and fill up the /var volume killing the vm.
>
>
> Diffs
> -----
>
> tools/appliance/definitions/systemvm64template/postinstall.sh cc8ead9
> tools/appliance/definitions/systemvmtemplate/postinstall.sh 23e66dd
>
> Diff: https://reviews.apache.org/r/21806/diff/
>
>
> Testing
> -------
>
> We are running this fix in our production environment for a couple months allready.
>
>
> Thanks,
>
> Joris van Lieshout
>
>
Re: Review Request 21806: Disable conntrackd-stats logging to prevent it
from filling up /var
Posted by Marcus Sorensen <sh...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21806/#review43710
-----------------------------------------------------------
I thought I had fixed this awhile back, but it was from the other direction. Conntrackd no longer restarts on log rotate unless conntrackd was already running (6b7f91d7). I suppose this wouldn't be bad to put in as well, so long as we know that conntrackd will in fact get started when we want it (redundant virtual router). Another issue contributing to it was that we had split up the tiny disk into many partitions, leaving only a few hundred MB available and stranded across many parts of the file tree, since there are times when we want conntrackd we should probably have a system vm template that can handle its logfiles.
You need to create a bug report that we can tie this to.
- Marcus Sorensen
On May 22, 2014, 11:42 a.m., Joris van Lieshout wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21806/
> -----------------------------------------------------------
>
> (Updated May 22, 2014, 11:42 a.m.)
>
>
> Review request for cloudstack, Abhinandan Prateek, daan Hoogland, Rajesh Battala, Rohit Yadav, and Hugo Trippaers.
>
>
> Repository: cloudstack-git
>
>
> Description
> -------
>
> Conntrackd package has a bug where the comment in the default config file states that stats logging is disabled by default but the config parameter is set to on. The consequence for ACS is that a conntrackd-stats.log file is created during the build of the svm. This logfile gets rotated by logrotate which has a post action to restart conntrackd. Even if the svm is not a redundant router. On vpc routers for instance the stats logging file can grow quickly and fill up the /var volume killing the vm.
>
>
> Diffs
> -----
>
> tools/appliance/definitions/systemvm64template/postinstall.sh cc8ead9
> tools/appliance/definitions/systemvmtemplate/postinstall.sh 23e66dd
>
> Diff: https://reviews.apache.org/r/21806/diff/
>
>
> Testing
> -------
>
> We are running this fix in our production environment for a couple months allready.
>
>
> Thanks,
>
> Joris van Lieshout
>
>