You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Serega Sheypak <se...@gmail.com> on 2016/05/10 09:32:21 UTC
storm 0.10, supervisor constantly dies when it tries to start task.
Hi, don't know what to do. What is the right debug the problem? Supervisor
constantly dies with provided execution on the same machine...
2016-05-06 23:12:15.271 b.s.event [ERROR] Error when processing event
java.lang.RuntimeException: org.apache.thrift7.protocol.TProtocolException:
Required field 'task_start' is unset! Struct:ExecutorInfo(task_start:0,
task_end:0)
at backtype.storm.utils.LocalState.deserialize(LocalState.java:83)
~[storm-core-0.10.0.jar:0.10.0]
at backtype.storm.utils.LocalState.get(LocalState.java:130)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.local_state$ls_local_assignments.invoke(local_state.clj:83)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:321)
~[storm-core-0.10.0.jar:0.10.0]
at clojure.lang.AFn.applyToHelper(AFn.java:154)
~[clojure-1.6.0.jar:?]
at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.6.0.jar:?]
at clojure.core$apply.invoke(core.clj:626) ~[clojure-1.6.0.jar:?]
at clojure.core$partial$fn__4228.doInvoke(core.clj:2468)
~[clojure-1.6.0.jar:?]
at clojure.lang.RestFn.invoke(RestFn.java:397) ~[clojure-1.6.0.jar:?]
at backtype.storm.event$event_manager$fn__7258.invoke(event.clj:40)
[storm-core-0.10.0.jar:0.10.0]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
Caused by: org.apache.thrift7.protocol.TProtocolException: Required field
'task_start' is unset! Struct:ExecutorInfo(task_start:0, task_end:0)
at
backtype.storm.generated.ExecutorInfo.validate(ExecutorInfo.java:386)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.generated.ExecutorInfo$ExecutorInfoStandardScheme.read(ExecutorInfo.java:454)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.generated.ExecutorInfo$ExecutorInfoStandardScheme.read(ExecutorInfo.java:420)
~[storm-core-0.10.0.jar:0.10.0]
at backtype.storm.generated.ExecutorInfo.read(ExecutorInfo.java:360)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.generated.LocalAssignment$LocalAssignmentStandardScheme.read(LocalAssignment.java:473)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.generated.LocalAssignment$LocalAssignmentStandardScheme.read(LocalAssignment.java:444)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.generated.LocalAssignment.read(LocalAssignment.java:378)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.generated.LSSupervisorAssignments$LSSupervisorAssignmentsStandardScheme.read(LSSupervisorAssignments.java:387)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.generated.LSSupervisorAssignments$LSSupervisorAssignmentsStandardScheme.read(LSSupervisorAssignments.java:364)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.generated.LSSupervisorAssignments.read(LSSupervisorAssignments.java:310)
~[storm-core-0.10.0.jar:0.10.0]
at
org.apache.thrift7.TDeserializer.deserialize(TDeserializer.java:81)
~[storm-core-0.10.0.jar:0.10.0]
at
org.apache.thrift7.TDeserializer.deserialize(TDeserializer.java:67)
~[storm-core-0.10.0.jar:0.10.0]
at backtype.storm.utils.LocalState.deserialize(LocalState.java:80)
~[storm-core-0.10.0.jar:0.10.0]
... 11 more
2016-05-06 23:12:15.278 b.s.util [ERROR] Halting process: ("Error when
processing an event")
java.lang.RuntimeException: ("Error when processing an event")
at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:336)
[storm-core-0.10.0.jar:0.10.0]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.6.0.jar:?]
at backtype.storm.event$event_manager$fn__7258.invoke(event.clj:48)
[storm-core-0.10.0.jar:0.10.0]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
2016-05-06 23:12:15.280 b.s.d.supervisor [INFO] Shutting down supervisor
290f8e54-6f7c-4f07-b4f6-06a5144f0772
2016-05-06 23:12:15.305 b.s.event [INFO] Event manager interrupted
Re: storm 0.10, supervisor constantly dies when it tries to start task.
Posted by Serega Sheypak <se...@gmail.com>.
Hi, that problem happens on the same node only. Supervisors on other hosts
don't have such problem.
rm /usr/lib/storm/storm-local/supervisor/localstate/*
Helps me to make supervisor work again.
2016-05-13 7:24 GMT+02:00 Rajasekhar <ra...@gmail.com>:
> Hi,
> Can you please share the code where you start the topology and topology
> configs.
>
> On Thu, May 12, 2016 at 1:41 PM, Serega Sheypak <se...@gmail.com>
> wrote:
>
>> Hi, could you please give some clue what it could be?
>>
>> 2016-05-10 11:32 GMT+02:00 Serega Sheypak <se...@gmail.com>:
>>
>>> Hi, don't know what to do. What is the right debug the problem?
>>> Supervisor constantly dies with provided execution on the same machine...
>>>
>>> 2016-05-06 23:12:15.271 b.s.event [ERROR] Error when processing event
>>> java.lang.RuntimeException:
>>> org.apache.thrift7.protocol.TProtocolException: Required field 'task_start'
>>> is unset! Struct:ExecutorInfo(task_start:0, task_end:0)
>>> at
>>> backtype.storm.utils.LocalState.deserialize(LocalState.java:83)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> at backtype.storm.utils.LocalState.get(LocalState.java:130)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> at
>>> backtype.storm.local_state$ls_local_assignments.invoke(local_state.clj:83)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> at
>>> backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:321)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> at clojure.lang.AFn.applyToHelper(AFn.java:154)
>>> ~[clojure-1.6.0.jar:?]
>>> at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.6.0.jar:?]
>>> at clojure.core$apply.invoke(core.clj:626) ~[clojure-1.6.0.jar:?]
>>> at clojure.core$partial$fn__4228.doInvoke(core.clj:2468)
>>> ~[clojure-1.6.0.jar:?]
>>> at clojure.lang.RestFn.invoke(RestFn.java:397)
>>> ~[clojure-1.6.0.jar:?]
>>> at
>>> backtype.storm.event$event_manager$fn__7258.invoke(event.clj:40)
>>> [storm-core-0.10.0.jar:0.10.0]
>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
>>> Caused by: org.apache.thrift7.protocol.TProtocolException: Required
>>> field 'task_start' is unset! Struct:ExecutorInfo(task_start:0, task_end:0)
>>> at
>>> backtype.storm.generated.ExecutorInfo.validate(ExecutorInfo.java:386)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> at
>>> backtype.storm.generated.ExecutorInfo$ExecutorInfoStandardScheme.read(ExecutorInfo.java:454)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> at
>>> backtype.storm.generated.ExecutorInfo$ExecutorInfoStandardScheme.read(ExecutorInfo.java:420)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> at
>>> backtype.storm.generated.ExecutorInfo.read(ExecutorInfo.java:360)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> at
>>> backtype.storm.generated.LocalAssignment$LocalAssignmentStandardScheme.read(LocalAssignment.java:473)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> at
>>> backtype.storm.generated.LocalAssignment$LocalAssignmentStandardScheme.read(LocalAssignment.java:444)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> at
>>> backtype.storm.generated.LocalAssignment.read(LocalAssignment.java:378)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> at
>>> backtype.storm.generated.LSSupervisorAssignments$LSSupervisorAssignmentsStandardScheme.read(LSSupervisorAssignments.java:387)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> at
>>> backtype.storm.generated.LSSupervisorAssignments$LSSupervisorAssignmentsStandardScheme.read(LSSupervisorAssignments.java:364)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> at
>>> backtype.storm.generated.LSSupervisorAssignments.read(LSSupervisorAssignments.java:310)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> at
>>> org.apache.thrift7.TDeserializer.deserialize(TDeserializer.java:81)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> at
>>> org.apache.thrift7.TDeserializer.deserialize(TDeserializer.java:67)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> at
>>> backtype.storm.utils.LocalState.deserialize(LocalState.java:80)
>>> ~[storm-core-0.10.0.jar:0.10.0]
>>> ... 11 more
>>> 2016-05-06 23:12:15.278 b.s.util [ERROR] Halting process: ("Error when
>>> processing an event")
>>> java.lang.RuntimeException: ("Error when processing an event")
>>> at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:336)
>>> [storm-core-0.10.0.jar:0.10.0]
>>> at clojure.lang.RestFn.invoke(RestFn.java:423)
>>> [clojure-1.6.0.jar:?]
>>> at
>>> backtype.storm.event$event_manager$fn__7258.invoke(event.clj:48)
>>> [storm-core-0.10.0.jar:0.10.0]
>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
>>> 2016-05-06 23:12:15.280 b.s.d.supervisor [INFO] Shutting down supervisor
>>> 290f8e54-6f7c-4f07-b4f6-06a5144f0772
>>> 2016-05-06 23:12:15.305 b.s.event [INFO] Event manager interrupted
>>>
>>
>>
>
>
> --
> Thanks & Regards
> Rajasekhar
>
Re: storm 0.10, supervisor constantly dies when it tries to start task.
Posted by Rajasekhar <ra...@gmail.com>.
Hi,
Can you please share the code where you start the topology and topology
configs.
On Thu, May 12, 2016 at 1:41 PM, Serega Sheypak <se...@gmail.com>
wrote:
> Hi, could you please give some clue what it could be?
>
> 2016-05-10 11:32 GMT+02:00 Serega Sheypak <se...@gmail.com>:
>
>> Hi, don't know what to do. What is the right debug the problem?
>> Supervisor constantly dies with provided execution on the same machine...
>>
>> 2016-05-06 23:12:15.271 b.s.event [ERROR] Error when processing event
>> java.lang.RuntimeException:
>> org.apache.thrift7.protocol.TProtocolException: Required field 'task_start'
>> is unset! Struct:ExecutorInfo(task_start:0, task_end:0)
>> at backtype.storm.utils.LocalState.deserialize(LocalState.java:83)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> at backtype.storm.utils.LocalState.get(LocalState.java:130)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> at
>> backtype.storm.local_state$ls_local_assignments.invoke(local_state.clj:83)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> at
>> backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:321)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> at clojure.lang.AFn.applyToHelper(AFn.java:154)
>> ~[clojure-1.6.0.jar:?]
>> at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.6.0.jar:?]
>> at clojure.core$apply.invoke(core.clj:626) ~[clojure-1.6.0.jar:?]
>> at clojure.core$partial$fn__4228.doInvoke(core.clj:2468)
>> ~[clojure-1.6.0.jar:?]
>> at clojure.lang.RestFn.invoke(RestFn.java:397)
>> ~[clojure-1.6.0.jar:?]
>> at
>> backtype.storm.event$event_manager$fn__7258.invoke(event.clj:40)
>> [storm-core-0.10.0.jar:0.10.0]
>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
>> Caused by: org.apache.thrift7.protocol.TProtocolException: Required field
>> 'task_start' is unset! Struct:ExecutorInfo(task_start:0, task_end:0)
>> at
>> backtype.storm.generated.ExecutorInfo.validate(ExecutorInfo.java:386)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> at
>> backtype.storm.generated.ExecutorInfo$ExecutorInfoStandardScheme.read(ExecutorInfo.java:454)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> at
>> backtype.storm.generated.ExecutorInfo$ExecutorInfoStandardScheme.read(ExecutorInfo.java:420)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> at
>> backtype.storm.generated.ExecutorInfo.read(ExecutorInfo.java:360)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> at
>> backtype.storm.generated.LocalAssignment$LocalAssignmentStandardScheme.read(LocalAssignment.java:473)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> at
>> backtype.storm.generated.LocalAssignment$LocalAssignmentStandardScheme.read(LocalAssignment.java:444)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> at
>> backtype.storm.generated.LocalAssignment.read(LocalAssignment.java:378)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> at
>> backtype.storm.generated.LSSupervisorAssignments$LSSupervisorAssignmentsStandardScheme.read(LSSupervisorAssignments.java:387)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> at
>> backtype.storm.generated.LSSupervisorAssignments$LSSupervisorAssignmentsStandardScheme.read(LSSupervisorAssignments.java:364)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> at
>> backtype.storm.generated.LSSupervisorAssignments.read(LSSupervisorAssignments.java:310)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> at
>> org.apache.thrift7.TDeserializer.deserialize(TDeserializer.java:81)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> at
>> org.apache.thrift7.TDeserializer.deserialize(TDeserializer.java:67)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> at backtype.storm.utils.LocalState.deserialize(LocalState.java:80)
>> ~[storm-core-0.10.0.jar:0.10.0]
>> ... 11 more
>> 2016-05-06 23:12:15.278 b.s.util [ERROR] Halting process: ("Error when
>> processing an event")
>> java.lang.RuntimeException: ("Error when processing an event")
>> at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:336)
>> [storm-core-0.10.0.jar:0.10.0]
>> at clojure.lang.RestFn.invoke(RestFn.java:423)
>> [clojure-1.6.0.jar:?]
>> at
>> backtype.storm.event$event_manager$fn__7258.invoke(event.clj:48)
>> [storm-core-0.10.0.jar:0.10.0]
>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
>> 2016-05-06 23:12:15.280 b.s.d.supervisor [INFO] Shutting down supervisor
>> 290f8e54-6f7c-4f07-b4f6-06a5144f0772
>> 2016-05-06 23:12:15.305 b.s.event [INFO] Event manager interrupted
>>
>
>
--
Thanks & Regards
Rajasekhar
Re: storm 0.10, supervisor constantly dies when it tries to start task.
Posted by Serega Sheypak <se...@gmail.com>.
Hi, could you please give some clue what it could be?
2016-05-10 11:32 GMT+02:00 Serega Sheypak <se...@gmail.com>:
> Hi, don't know what to do. What is the right debug the problem? Supervisor
> constantly dies with provided execution on the same machine...
>
> 2016-05-06 23:12:15.271 b.s.event [ERROR] Error when processing event
> java.lang.RuntimeException:
> org.apache.thrift7.protocol.TProtocolException: Required field 'task_start'
> is unset! Struct:ExecutorInfo(task_start:0, task_end:0)
> at backtype.storm.utils.LocalState.deserialize(LocalState.java:83)
> ~[storm-core-0.10.0.jar:0.10.0]
> at backtype.storm.utils.LocalState.get(LocalState.java:130)
> ~[storm-core-0.10.0.jar:0.10.0]
> at
> backtype.storm.local_state$ls_local_assignments.invoke(local_state.clj:83)
> ~[storm-core-0.10.0.jar:0.10.0]
> at
> backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:321)
> ~[storm-core-0.10.0.jar:0.10.0]
> at clojure.lang.AFn.applyToHelper(AFn.java:154)
> ~[clojure-1.6.0.jar:?]
> at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.6.0.jar:?]
> at clojure.core$apply.invoke(core.clj:626) ~[clojure-1.6.0.jar:?]
> at clojure.core$partial$fn__4228.doInvoke(core.clj:2468)
> ~[clojure-1.6.0.jar:?]
> at clojure.lang.RestFn.invoke(RestFn.java:397)
> ~[clojure-1.6.0.jar:?]
> at backtype.storm.event$event_manager$fn__7258.invoke(event.clj:40)
> [storm-core-0.10.0.jar:0.10.0]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
> Caused by: org.apache.thrift7.protocol.TProtocolException: Required field
> 'task_start' is unset! Struct:ExecutorInfo(task_start:0, task_end:0)
> at
> backtype.storm.generated.ExecutorInfo.validate(ExecutorInfo.java:386)
> ~[storm-core-0.10.0.jar:0.10.0]
> at
> backtype.storm.generated.ExecutorInfo$ExecutorInfoStandardScheme.read(ExecutorInfo.java:454)
> ~[storm-core-0.10.0.jar:0.10.0]
> at
> backtype.storm.generated.ExecutorInfo$ExecutorInfoStandardScheme.read(ExecutorInfo.java:420)
> ~[storm-core-0.10.0.jar:0.10.0]
> at
> backtype.storm.generated.ExecutorInfo.read(ExecutorInfo.java:360)
> ~[storm-core-0.10.0.jar:0.10.0]
> at
> backtype.storm.generated.LocalAssignment$LocalAssignmentStandardScheme.read(LocalAssignment.java:473)
> ~[storm-core-0.10.0.jar:0.10.0]
> at
> backtype.storm.generated.LocalAssignment$LocalAssignmentStandardScheme.read(LocalAssignment.java:444)
> ~[storm-core-0.10.0.jar:0.10.0]
> at
> backtype.storm.generated.LocalAssignment.read(LocalAssignment.java:378)
> ~[storm-core-0.10.0.jar:0.10.0]
> at
> backtype.storm.generated.LSSupervisorAssignments$LSSupervisorAssignmentsStandardScheme.read(LSSupervisorAssignments.java:387)
> ~[storm-core-0.10.0.jar:0.10.0]
> at
> backtype.storm.generated.LSSupervisorAssignments$LSSupervisorAssignmentsStandardScheme.read(LSSupervisorAssignments.java:364)
> ~[storm-core-0.10.0.jar:0.10.0]
> at
> backtype.storm.generated.LSSupervisorAssignments.read(LSSupervisorAssignments.java:310)
> ~[storm-core-0.10.0.jar:0.10.0]
> at
> org.apache.thrift7.TDeserializer.deserialize(TDeserializer.java:81)
> ~[storm-core-0.10.0.jar:0.10.0]
> at
> org.apache.thrift7.TDeserializer.deserialize(TDeserializer.java:67)
> ~[storm-core-0.10.0.jar:0.10.0]
> at backtype.storm.utils.LocalState.deserialize(LocalState.java:80)
> ~[storm-core-0.10.0.jar:0.10.0]
> ... 11 more
> 2016-05-06 23:12:15.278 b.s.util [ERROR] Halting process: ("Error when
> processing an event")
> java.lang.RuntimeException: ("Error when processing an event")
> at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:336)
> [storm-core-0.10.0.jar:0.10.0]
> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.6.0.jar:?]
> at backtype.storm.event$event_manager$fn__7258.invoke(event.clj:48)
> [storm-core-0.10.0.jar:0.10.0]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
> 2016-05-06 23:12:15.280 b.s.d.supervisor [INFO] Shutting down supervisor
> 290f8e54-6f7c-4f07-b4f6-06a5144f0772
> 2016-05-06 23:12:15.305 b.s.event [INFO] Event manager interrupted
>