You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by clay teahouse <cl...@gmail.com> on 2015/05/05 03:21:43 UTC
storm logback freezing
Hi all,
Has anyone experienced a case where storm logback freezes? The topology
seems to be functioning without an issue (I can see the results, in the
destination consumers), but the storm log shows no progress. This usually
happens a couple of hours after the topology starts, and not right away.
I'd appreciate any feedback that what may be causing this issue.
thank you
Clay
Re: storm logback freezing
Posted by clay teahouse <cl...@gmail.com>.
I get the log freezing even if I run the topology with 1 worker or if I run
the topology in local cluster node.
On Tue, May 5, 2015 at 11:00 AM, Jeff Maass <JM...@cccis.com> wrote:
> Here is one possibility:
>
> A worker will stop writing to its log if the supervisor kills it. As I
> know it, this happens when one of 2 things occur:
> * nimbus orders the worker shut down - after not hearing from an executor
> or after a manual topology kill is run
> * the supervisor doesn’t receive a heartbeat message from the worker
> process. This appears to happen when the worker is resource constrained -
> namely - CPU.
>
> In the second situation, very quickly after the worker is shutdown,
> nimbus tells a supervisor ( maybe the same one as previously ) to start up
> a worker, doing the same tasks. So, if you are watching the worker process
> running, you could say, walk away for a cup of coffee, and the worker could
> change from writing to worker_6702 to worker_6703.
>
> How to tell this has happened:
> * the worker pid will have changed
> * the worker’s communication port / node may have changed
> * nimbus logs - if nimbus initiated the action and the logging level is
> set to info
> * the supervisor logs - if the logging level is set to info
> * count and record the # of worker processes on your nodes
> * the storm metric - new worker events - has not to me proved to be
> accurate. I do not know why.
>
> From: clay teahouse <cl...@gmail.com>
> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
> Date: 2015,Tuesday, May 5 at 10:31
> To: "user@storm.apache.org" <us...@storm.apache.org>, Bobby Evans <
> evans@yahoo-inc.com>
> Subject: Re: storm logback freezing
>
> Hello,
> I am sorry for not being specific. The worker log is freezing. There is
> plenty of space on disk. I have made changes to logback config file, as
> follows.
> <configuration scan="true" scanPeriod="30 seconds">
> <appender name="A1" class="ch.qos.logback.core.ConsoleAppender">
> <encoder>
> <pattern>%-4r [%t] %-5p %c - %m%n</pattern>
> </encoder>
> </appender>
>
> <appender name="A2"
> class="ch.qos.logback.core.rolling.RollingFileAppender">
> <file>/usr/local/storm/log/storm.log</file>
> <rollingPolicy
> class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
>
> <fileNamePattern>/usr/local/storm/log/storm.log.%i.zip</fileNamePattern>
> <minIndex>1</minIndex>
> <maxIndex>5</maxIndex>
> </rollingPolicy>
>
> <triggeringPolicy
> class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
> <maxFileSize>10MB</maxFileSize>
> </triggeringPolicy>
> <append>true</append>
> <encoder>
> <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} -
> %msg%n</pattern>
> </encoder>
> </appender>
>
> <logger name="org.apache.zookeeper" level="WARN"/>
> <root level="INFO">
> <appender-ref ref="A2"/>
> </root>
>
> On Tue, May 5, 2015 at 8:34 AM, Bobby Evans <ev...@yahoo-inc.com> wrote:
>
>> The only time I have seen logback "freeze" was when the disk it was
>> writing to filled up. And in that case some log messages started to block,
>> and it was very painful.
>>
>> - Bobby
>>
>>
>>
>> On Tuesday, May 5, 2015 8:18 AM, Jeff Maass <JM...@cccis.com> wrote:
>>
>>
>> there are at least 3 different logs everyone should have:
>> * nimbus
>> * worker_{port}
>> * supervisor
>>
>> Which of these is “freezing”?
>>
>> What version storm?
>>
>> When a worker starts, does writing to its log work?
>>
>> Have you made any changes to your logback configuration?
>>
>>
>>
>> From: clay teahouse <cl...@gmail.com>
>> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
>> Date: 2015,Monday, May 4 at 20:21
>> To: "user@storm.incubator.apache.org" <us...@storm.incubator.apache.org>,
>> "dev@storm.incubator.apache.org" <de...@storm.incubator.apache.org>
>> Subject: storm logback freezing
>>
>> Hi all,
>> Has anyone experienced a case where storm logback freezes? The topology
>> seems to be functioning without an issue (I can see the results, in the
>> destination consumers), but the storm log shows no progress. This usually
>> happens a couple of hours after the topology starts, and not right away.
>> I'd appreciate any feedback that what may be causing this issue.
>>
>> thank you
>>
>> Clay
>>
>>
>>
>>
>
Re: storm logback freezing
Posted by Jeff Maass <JM...@cccis.com>.
Here is one possibility:
A worker will stop writing to its log if the supervisor kills it. As I know it, this happens when one of 2 things occur:
* nimbus orders the worker shut down - after not hearing from an executor or after a manual topology kill is run
* the supervisor doesn't receive a heartbeat message from the worker process. This appears to happen when the worker is resource constrained - namely - CPU.
In the second situation, very quickly after the worker is shutdown, nimbus tells a supervisor ( maybe the same one as previously ) to start up a worker, doing the same tasks. So, if you are watching the worker process running, you could say, walk away for a cup of coffee, and the worker could change from writing to worker_6702 to worker_6703.
How to tell this has happened:
* the worker pid will have changed
* the worker's communication port / node may have changed
* nimbus logs - if nimbus initiated the action and the logging level is set to info
* the supervisor logs - if the logging level is set to info
* count and record the # of worker processes on your nodes
* the storm metric - new worker events - has not to me proved to be accurate. I do not know why.
From: clay teahouse <cl...@gmail.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: 2015,Tuesday, May 5 at 10:31
To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>, Bobby Evans <ev...@yahoo-inc.com>>
Subject: Re: storm logback freezing
Hello,
I am sorry for not being specific. The worker log is freezing. There is plenty of space on disk. I have made changes to logback config file, as follows.
<configuration scan="true" scanPeriod="30 seconds">
<appender name="A1" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%-4r [%t] %-5p %c - %m%n</pattern>
</encoder>
</appender>
<appender name="A2" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>/usr/local/storm/log/storm.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>/usr/local/storm/log/storm.log.%i.zip</fileNamePattern>
<minIndex>1</minIndex>
<maxIndex>5</maxIndex>
</rollingPolicy>
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<maxFileSize>10MB</maxFileSize>
</triggeringPolicy>
<append>true</append>
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
</encoder>
</appender>
<logger name="org.apache.zookeeper" level="WARN"/>
<root level="INFO">
<appender-ref ref="A2"/>
</root>
On Tue, May 5, 2015 at 8:34 AM, Bobby Evans <ev...@yahoo-inc.com>> wrote:
The only time I have seen logback "freeze" was when the disk it was writing to filled up. And in that case some log messages started to block, and it was very painful.
- Bobby
On Tuesday, May 5, 2015 8:18 AM, Jeff Maass <JM...@cccis.com>> wrote:
there are at least 3 different logs everyone should have:
* nimbus
* worker_{port}
* supervisor
Which of these is "freezing"?
What version storm?
When a worker starts, does writing to its log work?
Have you made any changes to your logback configuration?
From: clay teahouse <cl...@gmail.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: 2015,Monday, May 4 at 20:21
To: "user@storm.incubator.apache.org<ma...@storm.incubator.apache.org>" <us...@storm.incubator.apache.org>>, "dev@storm.incubator.apache.org<ma...@storm.incubator.apache.org>" <de...@storm.incubator.apache.org>>
Subject: storm logback freezing
Hi all,
Has anyone experienced a case where storm logback freezes? The topology seems to be functioning without an issue (I can see the results, in the destination consumers), but the storm log shows no progress. This usually happens a couple of hours after the topology starts, and not right away. I'd appreciate any feedback that what may be causing this issue.
thank you
Clay
Re: storm logback freezing
Posted by clay teahouse <cl...@gmail.com>.
Hello,
I am sorry for not being specific. The worker log is freezing. There is
plenty of space on disk. I have made changes to logback config file, as
follows.
<configuration scan="true" scanPeriod="30 seconds">
<appender name="A1" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%-4r [%t] %-5p %c - %m%n</pattern>
</encoder>
</appender>
<appender name="A2"
class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>/usr/local/storm/log/storm.log</file>
<rollingPolicy
class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>/usr/local/storm/log/storm.log.%i.zip</fileNamePattern>
<minIndex>1</minIndex>
<maxIndex>5</maxIndex>
</rollingPolicy>
<triggeringPolicy
class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<maxFileSize>10MB</maxFileSize>
</triggeringPolicy>
<append>true</append>
<encoder>
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} -
%msg%n</pattern>
</encoder>
</appender>
<logger name="org.apache.zookeeper" level="WARN"/>
<root level="INFO">
<appender-ref ref="A2"/>
</root>
On Tue, May 5, 2015 at 8:34 AM, Bobby Evans <ev...@yahoo-inc.com> wrote:
> The only time I have seen logback "freeze" was when the disk it was
> writing to filled up. And in that case some log messages started to block,
> and it was very painful.
>
> - Bobby
>
>
>
> On Tuesday, May 5, 2015 8:18 AM, Jeff Maass <JM...@cccis.com> wrote:
>
>
> there are at least 3 different logs everyone should have:
> * nimbus
> * worker_{port}
> * supervisor
>
> Which of these is “freezing”?
>
> What version storm?
>
> When a worker starts, does writing to its log work?
>
> Have you made any changes to your logback configuration?
>
>
>
> From: clay teahouse <cl...@gmail.com>
> Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
> Date: 2015,Monday, May 4 at 20:21
> To: "user@storm.incubator.apache.org" <us...@storm.incubator.apache.org>, "
> dev@storm.incubator.apache.org" <de...@storm.incubator.apache.org>
> Subject: storm logback freezing
>
> Hi all,
> Has anyone experienced a case where storm logback freezes? The topology
> seems to be functioning without an issue (I can see the results, in the
> destination consumers), but the storm log shows no progress. This usually
> happens a couple of hours after the topology starts, and not right away.
> I'd appreciate any feedback that what may be causing this issue.
>
> thank you
>
> Clay
>
>
>
>
Re: storm logback freezing
Posted by Bobby Evans <ev...@yahoo-inc.com>.
The only time I have seen logback "freeze" was when the disk it was writing to filled up. And in that case some log messages started to block, and it was very painful.
- Bobby
On Tuesday, May 5, 2015 8:18 AM, Jeff Maass <JM...@cccis.com> wrote:
there are at least 3 different logs everyone should have:* nimbus* worker_{port}* supervisor
Which of these is “freezing”?
What version storm?
When a worker starts, does writing to its log work?
Have you made any changes to your logback configuration?
From: clay teahouse <cl...@gmail.com>
Reply-To: "user@storm.apache.org" <us...@storm.apache.org>
Date: 2015,Monday, May 4 at 20:21
To: "user@storm.incubator.apache.org" <us...@storm.incubator.apache.org>, "dev@storm.incubator.apache.org" <de...@storm.incubator.apache.org>
Subject: storm logback freezing
Hi all,Has anyone experienced a case where storm logback freezes? The topology seems to be functioning without an issue (I can see the results, in the destination consumers), but the storm log shows no progress. This usually happens a couple of hours after the topology starts, and not right away. I'd appreciate any feedback that what may be causing this issue.
thank you
Clay
Re: storm logback freezing
Posted by Jeff Maass <JM...@cccis.com>.
there are at least 3 different logs everyone should have:
* nimbus
* worker_{port}
* supervisor
Which of these is "freezing"?
What version storm?
When a worker starts, does writing to its log work?
Have you made any changes to your logback configuration?
From: clay teahouse <cl...@gmail.com>>
Reply-To: "user@storm.apache.org<ma...@storm.apache.org>" <us...@storm.apache.org>>
Date: 2015,Monday, May 4 at 20:21
To: "user@storm.incubator.apache.org<ma...@storm.incubator.apache.org>" <us...@storm.incubator.apache.org>>, "dev@storm.incubator.apache.org<ma...@storm.incubator.apache.org>" <de...@storm.incubator.apache.org>>
Subject: storm logback freezing
Hi all,
Has anyone experienced a case where storm logback freezes? The topology seems to be functioning without an issue (I can see the results, in the destination consumers), but the storm log shows no progress. This usually happens a couple of hours after the topology starts, and not right away. I'd appreciate any feedback that what may be causing this issue.
thank you
Clay