You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stratos.apache.org by "Jeffrey Nguyen (jeffrngu)" <je...@cisco.com> on 2014/06/28 01:52:23 UTC
Carbon log flooded causing disk full and reboot
Hi,
In our Stratos deployment, we experienced issue where wso2carbon.log file grew as much as 13 GB causing disk full and instance reboot and Stratos inaccessible. We're deploying Stratos using default profile. To fix this issue, we modified log4j.properties with the changes listed below.
Please let me know if it's okay to push this changes upstream. If so, which branch(es) should the changes be checked in?
Thanks,
-Jeffrey
diff --git a/products/stratos/modules/distribution/src/main/conf/log4j.properties b/products/stratos/modules/distribution/src
index a85d346..bb37679 100644
--- a/products/stratos/modules/distribution/src/main/conf/log4j.properties
+++ b/products/stratos/modules/distribution/src/main/conf/log4j.properties
@@ -113,7 +113,9 @@ log4j.appender.CARBON_MEMORY.threshold=DEBUG
# CARBON_LOGFILE is set to be a DailyRollingFileAppender using a PatternLayout.
-log4j.appender.CARBON_LOGFILE=org.apache.log4j.DailyRollingFileAppender
+log4j.appender.CARBON_LOGFILE=org.apache.log4j.RollingFileAppender
+log4j.appender.CARBON_LOGFILE.MaxFileSize=10000KB
+log4j.appender.CARBON_LOGFILE.MaxBackupIndex=10
# Log file will be overridden by the configuration setting in the DB
# This path should be relative to WSO2 Carbon Home
Re: Carbon log flooded causing disk full and reboot
Posted by "Jeffrey Nguyen (jeffrngu)" <je...@cisco.com>.
Thanks Imesh, Isuru, Amila and Martin.
As Martin mentioned, we had log enabled at DEBUG level. This change will additional level of protection besides having good logging instrumentation in the code. With the DailyRollingFileAppender, we can only hope that log file is rotated every 24 hr before the disk is filled up.
I've committed this change to both master and 4.0.0. Please feel free to revert or make adjustment to the size/number of files if you think necessary.
Regards,
-Jeffrey
From: Imesh Gunaratne <im...@apache.org>>
Reply-To: "dev@stratos.apache.org<ma...@stratos.apache.org>" <de...@stratos.apache.org>>
Date: Sunday, June 29, 2014 1:45 PM
To: dev <de...@stratos.apache.org>>
Subject: Re: Carbon log flooded causing disk full and reboot
Hi Jeffrey,
A great finding! Did this occur while running Stratos in single JVM mode without having any DEBUG level logs?
Please send these changes to both master and 4.0.0 branches.
Thanks
On Fri, Jun 27, 2014 at 7:52 PM, Jeffrey Nguyen (jeffrngu) <je...@cisco.com>> wrote:
Hi,
In our Stratos deployment, we experienced issue where wso2carbon.log file grew as much as 13 GB causing disk full and instance reboot and Stratos inaccessible. We're deploying Stratos using default profile. To fix this issue, we modified log4j.properties with the changes listed below.
Please let me know if it's okay to push this changes upstream. If so, which branch(es) should the changes be checked in?
Thanks,
-Jeffrey
diff --git a/products/stratos/modules/distribution/src/main/conf/log4j.properties b/products/stratos/modules/distribution/src
index a85d346..bb37679 100644
--- a/products/stratos/modules/distribution/src/main/conf/log4j.properties
+++ b/products/stratos/modules/distribution/src/main/conf/log4j.properties
@@ -113,7 +113,9 @@ log4j.appender.CARBON_MEMORY.threshold=DEBUG
# CARBON_LOGFILE is set to be a DailyRollingFileAppender using a PatternLayout.
-log4j.appender.CARBON_LOGFILE=org.apache.log4j.DailyRollingFileAppender
+log4j.appender.CARBON_LOGFILE=org.apache.log4j.RollingFileAppender
+log4j.appender.CARBON_LOGFILE.MaxFileSize=10000KB
+log4j.appender.CARBON_LOGFILE.MaxBackupIndex=10
# Log file will be overridden by the configuration setting in the DB
# This path should be relative to WSO2 Carbon Home
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PPMC Member, Apache Stratos
RE: Carbon log flooded causing disk full and reboot
Posted by "Martin Eppel (meppel)" <me...@cisco.com>.
Hi Amila,
I think debug logs were turned on and they contain additional debug statements of experimental features not yet in stratos.
Thanks
Martin
From: Amila Maha Arachchi [mailto:amilam@wso2.com]
Sent: Monday, June 30, 2014 5:09 AM
To: dev@stratos.apache.org
Subject: Re: Carbon log flooded causing disk full and reboot
I have seen a continuous log getting printed in the Message Broker of a Stratos setup I have (due to no subscribers available for a certain topic). But not in Stratos instance. For the MB logs (one is an INFO and the other is a WARN), I modified the log4j.properties to change log level of the relevant classes.
Jeffrey,
Can you post some of the frequently printed logs?
On Mon, Jun 30, 2014 at 12:16 PM, Isuru Perera <is...@wso2.com>> wrote:
We usually use daily rolling file appenders for log files. Did you check what logs were printed more frequently? Perhaps we should fix the code to change levels of log messages.
On Sun, Jun 29, 2014 at 1:45 PM, Imesh Gunaratne <im...@apache.org>> wrote:
Hi Jeffrey,
A great finding! Did this occur while running Stratos in single JVM mode without having any DEBUG level logs?
Please send these changes to both master and 4.0.0 branches.
Thanks
On Fri, Jun 27, 2014 at 7:52 PM, Jeffrey Nguyen (jeffrngu) <je...@cisco.com>> wrote:
Hi,
In our Stratos deployment, we experienced issue where wso2carbon.log file grew as much as 13 GB causing disk full and instance reboot and Stratos inaccessible. We're deploying Stratos using default profile. To fix this issue, we modified log4j.properties with the changes listed below.
Please let me know if it's okay to push this changes upstream. If so, which branch(es) should the changes be checked in?
Thanks,
-Jeffrey
diff --git a/products/stratos/modules/distribution/src/main/conf/log4j.properties b/products/stratos/modules/distribution/src
index a85d346..bb37679 100644
--- a/products/stratos/modules/distribution/src/main/conf/log4j.properties
+++ b/products/stratos/modules/distribution/src/main/conf/log4j.properties
@@ -113,7 +113,9 @@ log4j.appender.CARBON_MEMORY.threshold=DEBUG
# CARBON_LOGFILE is set to be a DailyRollingFileAppender using a PatternLayout.
-log4j.appender.CARBON_LOGFILE=org.apache.log4j.DailyRollingFileAppender
+log4j.appender.CARBON_LOGFILE=org.apache.log4j.RollingFileAppender
+log4j.appender.CARBON_LOGFILE.MaxFileSize=10000KB
+log4j.appender.CARBON_LOGFILE.MaxBackupIndex=10
# Log file will be overridden by the configuration setting in the DB
# This path should be relative to WSO2 Carbon Home
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PPMC Member, Apache Stratos
--
Isuru Perera
Senior Software Engineer | WSO2, Inc. | http://wso2.com/
Lean . Enterprise . Middleware
about.me/chrishantha<http://about.me/chrishantha>
--
Amila Maharachchi
Senior Technical Lead
WSO2, Inc.; http://wso2.com<http://wso2.com/>
Blog: http://maharachchi.blogspot.com
Mobile: +94719371446
Re: Carbon log flooded causing disk full and reboot
Posted by Amila Maha Arachchi <am...@wso2.com>.
I have seen a continuous log getting printed in the Message Broker of a
Stratos setup I have (due to no subscribers available for a certain topic).
But not in Stratos instance. For the MB logs (one is an INFO and the other
is a WARN), I modified the log4j.properties to change log level of the
relevant classes.
Jeffrey,
Can you post some of the frequently printed logs?
On Mon, Jun 30, 2014 at 12:16 PM, Isuru Perera <is...@wso2.com> wrote:
> We usually use daily rolling file appenders for log files. Did you check
> what logs were printed more frequently? Perhaps we should fix the code to
> change levels of log messages.
>
>
> On Sun, Jun 29, 2014 at 1:45 PM, Imesh Gunaratne <im...@apache.org> wrote:
>
>> Hi Jeffrey,
>>
>> A great finding! Did this occur while running Stratos in single JVM mode
>> without having any DEBUG level logs?
>> Please send these changes to both master and 4.0.0 branches.
>>
>> Thanks
>>
>>
>> On Fri, Jun 27, 2014 at 7:52 PM, Jeffrey Nguyen (jeffrngu) <
>> jeffrngu@cisco.com> wrote:
>>
>>> Hi,
>>>
>>> In our Stratos deployment, we experienced issue where wso2carbon.log
>>> file grew as much as 13 GB causing disk full and instance reboot and
>>> Stratos inaccessible. We're deploying Stratos using default profile. To
>>> fix this issue, we modified log4j.properties with the changes listed below.
>>>
>>> Please let me know if it's okay to push this changes upstream. If
>>> so, which branch(es) should the changes be checked in?
>>>
>>> Thanks,
>>> -Jeffrey
>>>
>>>
>>> diff --git
>>> a/products/stratos/modules/distribution/src/main/conf/log4j.properties
>>> b/products/stratos/modules/distribution/src
>>> index a85d346..bb37679 100644
>>> ---
>>> a/products/stratos/modules/distribution/src/main/conf/log4j.properties
>>> +++
>>> b/products/stratos/modules/distribution/src/main/conf/log4j.properties
>>> @@ -113,7 +113,9 @@ log4j.appender.CARBON_MEMORY.threshold=DEBUG
>>>
>>>
>>> # CARBON_LOGFILE is set to be a DailyRollingFileAppender using a
>>> PatternLayout.
>>> -log4j.appender.CARBON_LOGFILE=org.apache.log4j.DailyRollingFileAppender
>>> +log4j.appender.CARBON_LOGFILE=org.apache.log4j.RollingFileAppender
>>> +log4j.appender.CARBON_LOGFILE.MaxFileSize=10000KB
>>> +log4j.appender.CARBON_LOGFILE.MaxBackupIndex=10
>>> # Log file will be overridden by the configuration setting in the DB
>>> # This path should be relative to WSO2 Carbon Home
>>>
>>
>>
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PPMC Member, Apache Stratos
>>
>
>
>
> --
> Isuru Perera
> Senior Software Engineer | WSO2, Inc. | http://wso2.com/
> Lean . Enterprise . Middleware
>
> about.me/chrishantha
>
--
*Amila Maharachchi*
Senior Technical Lead
WSO2, Inc.; http://wso2.com
Blog: http://maharachchi.blogspot.com
Mobile: +94719371446
Re: Carbon log flooded causing disk full and reboot
Posted by Isuru Perera <is...@wso2.com>.
We usually use daily rolling file appenders for log files. Did you check
what logs were printed more frequently? Perhaps we should fix the code to
change levels of log messages.
On Sun, Jun 29, 2014 at 1:45 PM, Imesh Gunaratne <im...@apache.org> wrote:
> Hi Jeffrey,
>
> A great finding! Did this occur while running Stratos in single JVM mode
> without having any DEBUG level logs?
> Please send these changes to both master and 4.0.0 branches.
>
> Thanks
>
>
> On Fri, Jun 27, 2014 at 7:52 PM, Jeffrey Nguyen (jeffrngu) <
> jeffrngu@cisco.com> wrote:
>
>> Hi,
>>
>> In our Stratos deployment, we experienced issue where wso2carbon.log
>> file grew as much as 13 GB causing disk full and instance reboot and
>> Stratos inaccessible. We're deploying Stratos using default profile. To
>> fix this issue, we modified log4j.properties with the changes listed below.
>>
>> Please let me know if it's okay to push this changes upstream. If so,
>> which branch(es) should the changes be checked in?
>>
>> Thanks,
>> -Jeffrey
>>
>>
>> diff --git
>> a/products/stratos/modules/distribution/src/main/conf/log4j.properties
>> b/products/stratos/modules/distribution/src
>> index a85d346..bb37679 100644
>> --- a/products/stratos/modules/distribution/src/main/conf/log4j.properties
>> +++ b/products/stratos/modules/distribution/src/main/conf/log4j.properties
>> @@ -113,7 +113,9 @@ log4j.appender.CARBON_MEMORY.threshold=DEBUG
>>
>>
>> # CARBON_LOGFILE is set to be a DailyRollingFileAppender using a
>> PatternLayout.
>> -log4j.appender.CARBON_LOGFILE=org.apache.log4j.DailyRollingFileAppender
>> +log4j.appender.CARBON_LOGFILE=org.apache.log4j.RollingFileAppender
>> +log4j.appender.CARBON_LOGFILE.MaxFileSize=10000KB
>> +log4j.appender.CARBON_LOGFILE.MaxBackupIndex=10
>> # Log file will be overridden by the configuration setting in the DB
>> # This path should be relative to WSO2 Carbon Home
>>
>
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PPMC Member, Apache Stratos
>
--
Isuru Perera
Senior Software Engineer | WSO2, Inc. | http://wso2.com/
Lean . Enterprise . Middleware
about.me/chrishantha
Re: Carbon log flooded causing disk full and reboot
Posted by Imesh Gunaratne <im...@apache.org>.
Hi Jeffrey,
A great finding! Did this occur while running Stratos in single JVM mode
without having any DEBUG level logs?
Please send these changes to both master and 4.0.0 branches.
Thanks
On Fri, Jun 27, 2014 at 7:52 PM, Jeffrey Nguyen (jeffrngu) <
jeffrngu@cisco.com> wrote:
> Hi,
>
> In our Stratos deployment, we experienced issue where wso2carbon.log
> file grew as much as 13 GB causing disk full and instance reboot and
> Stratos inaccessible. We're deploying Stratos using default profile. To
> fix this issue, we modified log4j.properties with the changes listed below.
>
> Please let me know if it's okay to push this changes upstream. If so,
> which branch(es) should the changes be checked in?
>
> Thanks,
> -Jeffrey
>
>
> diff --git
> a/products/stratos/modules/distribution/src/main/conf/log4j.properties
> b/products/stratos/modules/distribution/src
> index a85d346..bb37679 100644
> --- a/products/stratos/modules/distribution/src/main/conf/log4j.properties
> +++ b/products/stratos/modules/distribution/src/main/conf/log4j.properties
> @@ -113,7 +113,9 @@ log4j.appender.CARBON_MEMORY.threshold=DEBUG
>
>
> # CARBON_LOGFILE is set to be a DailyRollingFileAppender using a
> PatternLayout.
> -log4j.appender.CARBON_LOGFILE=org.apache.log4j.DailyRollingFileAppender
> +log4j.appender.CARBON_LOGFILE=org.apache.log4j.RollingFileAppender
> +log4j.appender.CARBON_LOGFILE.MaxFileSize=10000KB
> +log4j.appender.CARBON_LOGFILE.MaxBackupIndex=10
> # Log file will be overridden by the configuration setting in the DB
> # This path should be relative to WSO2 Carbon Home
>
--
Imesh Gunaratne
Technical Lead, WSO2
Committer & PPMC Member, Apache Stratos