You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by aarthi ranganathan <aa...@gmail.com> on 2015/05/06 14:32:00 UTC
Storm jar transport error
Hi
We are using Storm Submitter via tomcat to submit topology to storm
cluster. The jar does not get uploaded to storm in the same machine as well
as a different machine. We are using apache-storm -0.9.4.This error occurs
intermittently. Following is the stack trace
java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:141)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:70)
at
com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.submitTopology(StormProcessor.java:148)
at
com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.access$000(StormProcessor.java:48)
at
com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor$1.call(StormProcessor.java:111)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
... 3 more
Caused by: org.apache.thrift7.transport.TTransportException
at
org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
at
org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101)
at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
at
org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
at
org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:69)
at
backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:162)
at backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:146)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:127)
... 8 more
We are not able to reproduce the issue for a particular scenario. It occurs
randomly. Can u throw some light on the issue? We have checked the firewall
settings and there is nothing blocked.
Thanks
Aarthi R
Re: Storm jar transport error
Posted by Erik Weathers <ew...@groupon.com>.
Did you try deleting the ZooKeeper state like I recommended?
- Erik
On Friday, May 8, 2015, aarthi ranganathan <aa...@gmail.com> wrote:
> We did a restart by deleting the entire storm local directory folder. The
> inbox folder is empty.
> On Fri, May 8, 2015 at 1:48 PM, bhargav sarvepalli <bond1king@gmail.com
> <javascript:_e(%7B%7D,'cvml','bond1king@gmail.com');>> wrote:
>
>> Can you check the contents of inbox folder? and probably clean it
>>
>> On Fri, May 8, 2015 at 1:11 PM, aarthi ranganathan <
>> aarthirangan@gmail.com
>> <javascript:_e(%7B%7D,'cvml','aarthirangan@gmail.com');>> wrote:
>>
>>> I tried clearing the local state and also restarted storm. Still
>>> getting Illegal Argument Exception. Since jar is not getting uploaded due
>>> to illegal argument exception, the topology does not exist in the zookeeper.
>>> The jar is getting uploaded from tomcat to nimbus inbox. Its not getting
>>> copied to storm dist folder. Im posting the recent stack trace where look
>>> into highlighted parts in bold. The jar uploaded starts with <stormjar-e7>
>>> while the jar that is trying to get transferred is <stormjar-f8>. Is storm
>>> caching old jars and trying to run? Our team also saw this storm issue
>>> https://issues.apache.org/jira/browse/STORM-467 which is still
>>> unresolved where they mention trying to execute non-existent jars as the
>>> issue.
>>>
>>>
>>> 2015-05-08T11:58:36.122+0530 b.s.d.nimbus [INFO] Uploading file from
>>> client to
>>> /home/xbdev/storm/nimbus/inbox/stormjar-e7c13d9a-3aed-4232-9d20-6813e9c45c03.jar
>>>
>>> 2015-05-08T11:58:38.546+0530 b.s.d.nimbus [INFO] Finished uploading file
>>> from client:
>>> */home/xbdev/storm/nimbus/inbox/stormjar-e7c13d9a-3aed-4232-9d20-6813e9c45c03.jar*
>>>
>>> 2015-05-08T11:58:38.710+0530 b.s.d.nimbus [INFO] Received topology
>>> submission for topology-1431056498295 with conf {"storm.id"
>>> "topology-1431056498295-1-1431066518", "cassandra-config" {"cassandra.host"
>>> "10.237.247.13", "cassandra.keyspace" ["stormks"]}, "nimbus.host"
>>> "10.237.247.3", "storm.zookeeper.port" 2181, "storm.zookeeper.servers"
>>> ["10.237.247.3"], "topology.acker.executors" nil,
>>> "topology.kryo.decorators" (), "topology.name"
>>> "topology-1431056498295", "topology.debug" true, "topology.kryo.register"
>>> nil, "storm.elasticsearch.hosts" "10.237.247.13:9300", "
>>> storm.elasticsearch.cluster.name" "HAP_Development_1.0",
>>> "topology.workers" 2, "nimbus.thrift.port" 6627, "storm.thrift.transport"
>>> "backtype.storm.security.auth.SimpleTransportPlugin",
>>> "topology.max.task.parallelism" nil}
>>>
>>> 2015-05-08T11:58:38.716+0530 b.s.d.nimbus [WARN] Topology submission
>>> exception. (topology name='topology-1431056498295')
>>> #<IllegalArgumentException java.lang.IllegalArgumentException:
>>> */home/xbdev/storm/nimbus/inbox/stormjar-f8982d19-81d0-420a-a847-7243eb6d4181.jar*
>>> to copy to
>>> /home/xbdev/storm/nimbus/stormdist/topology-1431056498295-1-1431066518 does
>>> not exist!>
>>>
>>> 2015-05-08T11:58:38.722+0530 o.a.t.s.TNonblockingServer [ERROR]
>>> Unexpected exception while invoking!
>>>
>>> java.lang.IllegalArgumentException:
>>> /home/xbdev/storm/nimbus/inbox/stormjar-f8982d19-81d0-420a-a847-7243eb6d4181.jar
>>> to copy to
>>> /home/xbdev/storm/nimbus/stormdist/topology-1431056498295-1-1431066518 does
>>> not exist!
>>>
>>> at backtype.storm.daemon.nimbus$fn__4393.invoke(nimbus.clj:1173)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>
>>> at clojure.lang.MultiFn.invoke(MultiFn.java:236) ~[clojure-1.5.1.jar:na]
>>>
>>> at backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:307)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>
>>> at
>>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopologyWithOpts(nimbus.clj:953)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>
>>> at
>>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopology(nimbus.clj:966)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>
>>> at
>>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1240)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>
>>> at
>>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1228)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>
>>> at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>
>>> at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>
>>> at
>>> org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>
>>> at
>>> org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201)
>>> [storm-core-0.9.4.jar:0.9.4]
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> [na:1.7.0_45]
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> [na:1.7.0_45]
>>>
>>> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
>>>
>>>
>>> On Fri, May 8, 2015 at 11:49 AM, Erik Weathers <eweathers@groupon.com
>>> <javascript:_e(%7B%7D,'cvml','eweathers@groupon.com');>> wrote:
>>>
>>>> If that doesn't work, then you might need to remove the state in
>>>> zookeeper related to this topology.
>>>> i.e.,
>>>>
>>>> - /storm/storms/<topology-id>
>>>> - /storm/assignments/<topology-id>
>>>>
>>>> - Erik
>>>>
>>>> On Thu, May 7, 2015 at 11:18 PM, bhargav sarvepalli <
>>>> bond1king@gmail.com
>>>> <javascript:_e(%7B%7D,'cvml','bond1king@gmail.com');>> wrote:
>>>>
>>>>> Try deleting your local state folder in supervisors and storm nimbus
>>>>> data folder, looks like its trying to find topology which does not exist.
>>>>>
>>>>> On Fri, May 8, 2015 at 11:08 AM, aarthi ranganathan <
>>>>> aarthirangan@gmail.com
>>>>> <javascript:_e(%7B%7D,'cvml','aarthirangan@gmail.com');>> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> While checking nimbus log, Im getting the following stack track
>>>>>> trace.. The previous stack trace was while running from tomcat application
>>>>>> , which generates an illegal argument exception in nimbus logs
>>>>>>
>>>>>> 2015-05-06T14:27:57.718+0530 o.a.t.s.TNonblockingServer [ERROR]
>>>>>> Unexpected exception while invoking!
>>>>>> java.lang.IllegalArgumentException:
>>>>>> /home/xbdev/storm/nimbus/inbox/stormjar-e9973a4f-7dc2-4cd5-a1ef-4e8c95ec9a34.jar
>>>>>> to copy to
>>>>>> /home/xbdev/storm/nimbus/stormdist/topology-1430892623717-3-1430902677 does
>>>>>> not exist!
>>>>>> at backtype.storm.daemon.nimbus$fn__4393.invoke(nimbus.clj:1173)
>>>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>>>> at clojure.lang.MultiFn.invoke(MultiFn.java:236)
>>>>>> ~[clojure-1.5.1.jar:na]
>>>>>> at
>>>>>> backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:307)
>>>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>>>> at
>>>>>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopologyWithOpts(nimbus.clj:953)
>>>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>>>> at
>>>>>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopology(nimbus.clj:966)
>>>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>>>> at
>>>>>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1240)
>>>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>>>> at
>>>>>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1228)
>>>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>>>> at
>>>>>> org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32)
>>>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>>>> at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34)
>>>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>>>> at
>>>>>> org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632)
>>>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>>>> at
>>>>>> org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201)
>>>>>> [storm-core-0.9.4.jar:0.9.4]
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> [na:1.7.0_45]
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> [na:1.7.0_45]
>>>>>> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, May 7, 2015 at 5:36 PM, Nishu <nishutayal@gmail.com
>>>>>> <javascript:_e(%7B%7D,'cvml','nishutayal@gmail.com');>> wrote:
>>>>>>
>>>>>>> You can check if there are free workers available to run the
>>>>>>> topology or not.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Nishu Tayal
>>>>>>> On 7 May 2015 17:08, "bhargav sarvepalli" <bond1king@gmail.com
>>>>>>> <javascript:_e(%7B%7D,'cvml','bond1king@gmail.com');>> wrote:
>>>>>>>
>>>>>>>> Your nimbus has gone down, probably because there is no space left
>>>>>>>> in your machine.
>>>>>>>>
>>>>>>>> On Thu, May 7, 2015 at 3:25 PM, aarthi ranganathan <
>>>>>>>> aarthirangan@gmail.com
>>>>>>>> <javascript:_e(%7B%7D,'cvml','aarthirangan@gmail.com');>> wrote:
>>>>>>>>
>>>>>>>>> The zookeeper exists in the same machine along with storm. So I
>>>>>>>>> guess the ports are accessible.
>>>>>>>>>
>>>>>>>>> On Wed, May 6, 2015 at 6:44 PM, Aditya Rajan <
>>>>>>>>> aditya.rajan@whizdm.com
>>>>>>>>> <javascript:_e(%7B%7D,'cvml','aditya.rajan@whizdm.com');>> wrote:
>>>>>>>>>
>>>>>>>>>> I would assume it's an issue with zookeeper. Have you checked if
>>>>>>>>>> the relevant ports are open?
>>>>>>>>>> On May 6, 2015 6:02 PM, "aarthi ranganathan" <
>>>>>>>>>> aarthirangan@gmail.com
>>>>>>>>>> <javascript:_e(%7B%7D,'cvml','aarthirangan@gmail.com');>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi
>>>>>>>>>>>
>>>>>>>>>>> We are using Storm Submitter via tomcat to submit topology to
>>>>>>>>>>> storm cluster. The jar does not get uploaded to storm in the same machine
>>>>>>>>>>> as well as a different machine. We are using apache-storm -0.9.4.This error
>>>>>>>>>>> occurs intermittently. Following is the stack trace
>>>>>>>>>>>
>>>>>>>>>>> java.lang.RuntimeException:
>>>>>>>>>>> org.apache.thrift7.transport.TTransportException
>>>>>>>>>>> at
>>>>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:141)
>>>>>>>>>>> at
>>>>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:70)
>>>>>>>>>>> at
>>>>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.submitTopology(StormProcessor.java:148)
>>>>>>>>>>> at
>>>>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.access$000(StormProcessor.java:48)
>>>>>>>>>>> at
>>>>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor$1.call(StormProcessor.java:111)
>>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>>>>>> ... 3 more
>>>>>>>>>>> Caused by: org.apache.thrift7.transport.TTransportException
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:69)
>>>>>>>>>>> at
>>>>>>>>>>> backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:162)
>>>>>>>>>>> at
>>>>>>>>>>> backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:146)
>>>>>>>>>>> at
>>>>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:127)
>>>>>>>>>>> ... 8 more
>>>>>>>>>>>
>>>>>>>>>>> We are not able to reproduce the issue for a particular
>>>>>>>>>>> scenario. It occurs randomly. Can u throw some light on the issue? We have
>>>>>>>>>>> checked the firewall settings and there is nothing blocked.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Aarthi R
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Aarthi R
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Aarthi R
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Aarthi R
>>>
>>
>>
>
>
> --
> Aarthi R
>
Re: Storm jar transport error
Posted by aarthi ranganathan <aa...@gmail.com>.
We did a restart by deleting the entire storm local directory folder. The
inbox folder is empty.
On Fri, May 8, 2015 at 1:48 PM, bhargav sarvepalli <bo...@gmail.com>
wrote:
> Can you check the contents of inbox folder? and probably clean it
>
> On Fri, May 8, 2015 at 1:11 PM, aarthi ranganathan <aarthirangan@gmail.com
> > wrote:
>
>> I tried clearing the local state and also restarted storm. Still getting
>> Illegal Argument Exception. Since jar is not getting uploaded due to
>> illegal argument exception, the topology does not exist in the zookeeper.
>> The jar is getting uploaded from tomcat to nimbus inbox. Its not getting
>> copied to storm dist folder. Im posting the recent stack trace where look
>> into highlighted parts in bold. The jar uploaded starts with <stormjar-e7>
>> while the jar that is trying to get transferred is <stormjar-f8>. Is storm
>> caching old jars and trying to run? Our team also saw this storm issue
>> https://issues.apache.org/jira/browse/STORM-467 which is still
>> unresolved where they mention trying to execute non-existent jars as the
>> issue.
>>
>>
>> 2015-05-08T11:58:36.122+0530 b.s.d.nimbus [INFO] Uploading file from
>> client to
>> /home/xbdev/storm/nimbus/inbox/stormjar-e7c13d9a-3aed-4232-9d20-6813e9c45c03.jar
>>
>> 2015-05-08T11:58:38.546+0530 b.s.d.nimbus [INFO] Finished uploading file
>> from client:
>> */home/xbdev/storm/nimbus/inbox/stormjar-e7c13d9a-3aed-4232-9d20-6813e9c45c03.jar*
>>
>> 2015-05-08T11:58:38.710+0530 b.s.d.nimbus [INFO] Received topology
>> submission for topology-1431056498295 with conf {"storm.id"
>> "topology-1431056498295-1-1431066518", "cassandra-config" {"cassandra.host"
>> "10.237.247.13", "cassandra.keyspace" ["stormks"]}, "nimbus.host"
>> "10.237.247.3", "storm.zookeeper.port" 2181, "storm.zookeeper.servers"
>> ["10.237.247.3"], "topology.acker.executors" nil,
>> "topology.kryo.decorators" (), "topology.name" "topology-1431056498295",
>> "topology.debug" true, "topology.kryo.register" nil,
>> "storm.elasticsearch.hosts" "10.237.247.13:9300", "
>> storm.elasticsearch.cluster.name" "HAP_Development_1.0",
>> "topology.workers" 2, "nimbus.thrift.port" 6627, "storm.thrift.transport"
>> "backtype.storm.security.auth.SimpleTransportPlugin",
>> "topology.max.task.parallelism" nil}
>>
>> 2015-05-08T11:58:38.716+0530 b.s.d.nimbus [WARN] Topology submission
>> exception. (topology name='topology-1431056498295')
>> #<IllegalArgumentException java.lang.IllegalArgumentException:
>> */home/xbdev/storm/nimbus/inbox/stormjar-f8982d19-81d0-420a-a847-7243eb6d4181.jar*
>> to copy to
>> /home/xbdev/storm/nimbus/stormdist/topology-1431056498295-1-1431066518 does
>> not exist!>
>>
>> 2015-05-08T11:58:38.722+0530 o.a.t.s.TNonblockingServer [ERROR]
>> Unexpected exception while invoking!
>>
>> java.lang.IllegalArgumentException:
>> /home/xbdev/storm/nimbus/inbox/stormjar-f8982d19-81d0-420a-a847-7243eb6d4181.jar
>> to copy to
>> /home/xbdev/storm/nimbus/stormdist/topology-1431056498295-1-1431066518 does
>> not exist!
>>
>> at backtype.storm.daemon.nimbus$fn__4393.invoke(nimbus.clj:1173)
>> ~[storm-core-0.9.4.jar:0.9.4]
>>
>> at clojure.lang.MultiFn.invoke(MultiFn.java:236) ~[clojure-1.5.1.jar:na]
>>
>> at backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:307)
>> ~[storm-core-0.9.4.jar:0.9.4]
>>
>> at
>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopologyWithOpts(nimbus.clj:953)
>> ~[storm-core-0.9.4.jar:0.9.4]
>>
>> at
>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopology(nimbus.clj:966)
>> ~[storm-core-0.9.4.jar:0.9.4]
>>
>> at
>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1240)
>> ~[storm-core-0.9.4.jar:0.9.4]
>>
>> at
>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1228)
>> ~[storm-core-0.9.4.jar:0.9.4]
>>
>> at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32)
>> ~[storm-core-0.9.4.jar:0.9.4]
>>
>> at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34)
>> ~[storm-core-0.9.4.jar:0.9.4]
>>
>> at
>> org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632)
>> ~[storm-core-0.9.4.jar:0.9.4]
>>
>> at
>> org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201)
>> [storm-core-0.9.4.jar:0.9.4]
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> [na:1.7.0_45]
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> [na:1.7.0_45]
>>
>> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
>>
>>
>> On Fri, May 8, 2015 at 11:49 AM, Erik Weathers <ew...@groupon.com>
>> wrote:
>>
>>> If that doesn't work, then you might need to remove the state in
>>> zookeeper related to this topology.
>>> i.e.,
>>>
>>> - /storm/storms/<topology-id>
>>> - /storm/assignments/<topology-id>
>>>
>>> - Erik
>>>
>>> On Thu, May 7, 2015 at 11:18 PM, bhargav sarvepalli <bond1king@gmail.com
>>> > wrote:
>>>
>>>> Try deleting your local state folder in supervisors and storm nimbus
>>>> data folder, looks like its trying to find topology which does not exist.
>>>>
>>>> On Fri, May 8, 2015 at 11:08 AM, aarthi ranganathan <
>>>> aarthirangan@gmail.com> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> While checking nimbus log, Im getting the following stack track
>>>>> trace.. The previous stack trace was while running from tomcat application
>>>>> , which generates an illegal argument exception in nimbus logs
>>>>>
>>>>> 2015-05-06T14:27:57.718+0530 o.a.t.s.TNonblockingServer [ERROR]
>>>>> Unexpected exception while invoking!
>>>>> java.lang.IllegalArgumentException:
>>>>> /home/xbdev/storm/nimbus/inbox/stormjar-e9973a4f-7dc2-4cd5-a1ef-4e8c95ec9a34.jar
>>>>> to copy to
>>>>> /home/xbdev/storm/nimbus/stormdist/topology-1430892623717-3-1430902677 does
>>>>> not exist!
>>>>> at backtype.storm.daemon.nimbus$fn__4393.invoke(nimbus.clj:1173)
>>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>>> at clojure.lang.MultiFn.invoke(MultiFn.java:236)
>>>>> ~[clojure-1.5.1.jar:na]
>>>>> at
>>>>> backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:307)
>>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>>> at
>>>>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopologyWithOpts(nimbus.clj:953)
>>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>>> at
>>>>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopology(nimbus.clj:966)
>>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>>> at
>>>>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1240)
>>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>>> at
>>>>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1228)
>>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>>> at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32)
>>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>>> at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34)
>>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>>> at
>>>>> org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632)
>>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>>> at
>>>>> org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201)
>>>>> [storm-core-0.9.4.jar:0.9.4]
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> [na:1.7.0_45]
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> [na:1.7.0_45]
>>>>> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, May 7, 2015 at 5:36 PM, Nishu <ni...@gmail.com> wrote:
>>>>>
>>>>>> You can check if there are free workers available to run the topology
>>>>>> or not.
>>>>>>
>>>>>> Thanks,
>>>>>> Nishu Tayal
>>>>>> On 7 May 2015 17:08, "bhargav sarvepalli" <bo...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Your nimbus has gone down, probably because there is no space left
>>>>>>> in your machine.
>>>>>>>
>>>>>>> On Thu, May 7, 2015 at 3:25 PM, aarthi ranganathan <
>>>>>>> aarthirangan@gmail.com> wrote:
>>>>>>>
>>>>>>>> The zookeeper exists in the same machine along with storm. So I
>>>>>>>> guess the ports are accessible.
>>>>>>>>
>>>>>>>> On Wed, May 6, 2015 at 6:44 PM, Aditya Rajan <
>>>>>>>> aditya.rajan@whizdm.com> wrote:
>>>>>>>>
>>>>>>>>> I would assume it's an issue with zookeeper. Have you checked if
>>>>>>>>> the relevant ports are open?
>>>>>>>>> On May 6, 2015 6:02 PM, "aarthi ranganathan" <
>>>>>>>>> aarthirangan@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> We are using Storm Submitter via tomcat to submit topology to
>>>>>>>>>> storm cluster. The jar does not get uploaded to storm in the same machine
>>>>>>>>>> as well as a different machine. We are using apache-storm -0.9.4.This error
>>>>>>>>>> occurs intermittently. Following is the stack trace
>>>>>>>>>>
>>>>>>>>>> java.lang.RuntimeException:
>>>>>>>>>> org.apache.thrift7.transport.TTransportException
>>>>>>>>>> at
>>>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:141)
>>>>>>>>>> at
>>>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:70)
>>>>>>>>>> at
>>>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.submitTopology(StormProcessor.java:148)
>>>>>>>>>> at
>>>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.access$000(StormProcessor.java:48)
>>>>>>>>>> at
>>>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor$1.call(StormProcessor.java:111)
>>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>>>>> ... 3 more
>>>>>>>>>> Caused by: org.apache.thrift7.transport.TTransportException
>>>>>>>>>> at
>>>>>>>>>> org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>>>>>>>>>> at
>>>>>>>>>> org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
>>>>>>>>>> at
>>>>>>>>>> org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
>>>>>>>>>> at
>>>>>>>>>> org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101)
>>>>>>>>>> at
>>>>>>>>>> org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
>>>>>>>>>> at
>>>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
>>>>>>>>>> at
>>>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
>>>>>>>>>> at
>>>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
>>>>>>>>>> at
>>>>>>>>>> org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:69)
>>>>>>>>>> at
>>>>>>>>>> backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:162)
>>>>>>>>>> at
>>>>>>>>>> backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:146)
>>>>>>>>>> at
>>>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:127)
>>>>>>>>>> ... 8 more
>>>>>>>>>>
>>>>>>>>>> We are not able to reproduce the issue for a particular scenario.
>>>>>>>>>> It occurs randomly. Can u throw some light on the issue? We have checked
>>>>>>>>>> the firewall settings and there is nothing blocked.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Aarthi R
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Aarthi R
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Aarthi R
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Aarthi R
>>
>
>
--
Aarthi R
Re: Storm jar transport error
Posted by bhargav sarvepalli <bo...@gmail.com>.
Can you check the contents of inbox folder? and probably clean it
On Fri, May 8, 2015 at 1:11 PM, aarthi ranganathan <aa...@gmail.com>
wrote:
> I tried clearing the local state and also restarted storm. Still getting
> Illegal Argument Exception. Since jar is not getting uploaded due to
> illegal argument exception, the topology does not exist in the zookeeper.
> The jar is getting uploaded from tomcat to nimbus inbox. Its not getting
> copied to storm dist folder. Im posting the recent stack trace where look
> into highlighted parts in bold. The jar uploaded starts with <stormjar-e7>
> while the jar that is trying to get transferred is <stormjar-f8>. Is storm
> caching old jars and trying to run? Our team also saw this storm issue
> https://issues.apache.org/jira/browse/STORM-467 which is still unresolved
> where they mention trying to execute non-existent jars as the issue.
>
>
> 2015-05-08T11:58:36.122+0530 b.s.d.nimbus [INFO] Uploading file from
> client to
> /home/xbdev/storm/nimbus/inbox/stormjar-e7c13d9a-3aed-4232-9d20-6813e9c45c03.jar
>
> 2015-05-08T11:58:38.546+0530 b.s.d.nimbus [INFO] Finished uploading file
> from client:
> */home/xbdev/storm/nimbus/inbox/stormjar-e7c13d9a-3aed-4232-9d20-6813e9c45c03.jar*
>
> 2015-05-08T11:58:38.710+0530 b.s.d.nimbus [INFO] Received topology
> submission for topology-1431056498295 with conf {"storm.id"
> "topology-1431056498295-1-1431066518", "cassandra-config" {"cassandra.host"
> "10.237.247.13", "cassandra.keyspace" ["stormks"]}, "nimbus.host"
> "10.237.247.3", "storm.zookeeper.port" 2181, "storm.zookeeper.servers"
> ["10.237.247.3"], "topology.acker.executors" nil,
> "topology.kryo.decorators" (), "topology.name" "topology-1431056498295",
> "topology.debug" true, "topology.kryo.register" nil,
> "storm.elasticsearch.hosts" "10.237.247.13:9300", "
> storm.elasticsearch.cluster.name" "HAP_Development_1.0",
> "topology.workers" 2, "nimbus.thrift.port" 6627, "storm.thrift.transport"
> "backtype.storm.security.auth.SimpleTransportPlugin",
> "topology.max.task.parallelism" nil}
>
> 2015-05-08T11:58:38.716+0530 b.s.d.nimbus [WARN] Topology submission
> exception. (topology name='topology-1431056498295')
> #<IllegalArgumentException java.lang.IllegalArgumentException:
> */home/xbdev/storm/nimbus/inbox/stormjar-f8982d19-81d0-420a-a847-7243eb6d4181.jar*
> to copy to
> /home/xbdev/storm/nimbus/stormdist/topology-1431056498295-1-1431066518 does
> not exist!>
>
> 2015-05-08T11:58:38.722+0530 o.a.t.s.TNonblockingServer [ERROR] Unexpected
> exception while invoking!
>
> java.lang.IllegalArgumentException:
> /home/xbdev/storm/nimbus/inbox/stormjar-f8982d19-81d0-420a-a847-7243eb6d4181.jar
> to copy to
> /home/xbdev/storm/nimbus/stormdist/topology-1431056498295-1-1431066518 does
> not exist!
>
> at backtype.storm.daemon.nimbus$fn__4393.invoke(nimbus.clj:1173)
> ~[storm-core-0.9.4.jar:0.9.4]
>
> at clojure.lang.MultiFn.invoke(MultiFn.java:236) ~[clojure-1.5.1.jar:na]
>
> at backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:307)
> ~[storm-core-0.9.4.jar:0.9.4]
>
> at
> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopologyWithOpts(nimbus.clj:953)
> ~[storm-core-0.9.4.jar:0.9.4]
>
> at
> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopology(nimbus.clj:966)
> ~[storm-core-0.9.4.jar:0.9.4]
>
> at
> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1240)
> ~[storm-core-0.9.4.jar:0.9.4]
>
> at
> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1228)
> ~[storm-core-0.9.4.jar:0.9.4]
>
> at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32)
> ~[storm-core-0.9.4.jar:0.9.4]
>
> at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34)
> ~[storm-core-0.9.4.jar:0.9.4]
>
> at
> org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632)
> ~[storm-core-0.9.4.jar:0.9.4]
>
> at
> org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201)
> [storm-core-0.9.4.jar:0.9.4]
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_45]
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_45]
>
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
>
>
> On Fri, May 8, 2015 at 11:49 AM, Erik Weathers <ew...@groupon.com>
> wrote:
>
>> If that doesn't work, then you might need to remove the state in
>> zookeeper related to this topology.
>> i.e.,
>>
>> - /storm/storms/<topology-id>
>> - /storm/assignments/<topology-id>
>>
>> - Erik
>>
>> On Thu, May 7, 2015 at 11:18 PM, bhargav sarvepalli <bo...@gmail.com>
>> wrote:
>>
>>> Try deleting your local state folder in supervisors and storm nimbus
>>> data folder, looks like its trying to find topology which does not exist.
>>>
>>> On Fri, May 8, 2015 at 11:08 AM, aarthi ranganathan <
>>> aarthirangan@gmail.com> wrote:
>>>
>>>> Hi
>>>>
>>>> While checking nimbus log, Im getting the following stack track trace..
>>>> The previous stack trace was while running from tomcat application , which
>>>> generates an illegal argument exception in nimbus logs
>>>>
>>>> 2015-05-06T14:27:57.718+0530 o.a.t.s.TNonblockingServer [ERROR]
>>>> Unexpected exception while invoking!
>>>> java.lang.IllegalArgumentException:
>>>> /home/xbdev/storm/nimbus/inbox/stormjar-e9973a4f-7dc2-4cd5-a1ef-4e8c95ec9a34.jar
>>>> to copy to
>>>> /home/xbdev/storm/nimbus/stormdist/topology-1430892623717-3-1430902677 does
>>>> not exist!
>>>> at backtype.storm.daemon.nimbus$fn__4393.invoke(nimbus.clj:1173)
>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>> at clojure.lang.MultiFn.invoke(MultiFn.java:236) ~[clojure-1.5.1.jar:na]
>>>> at backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:307)
>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>> at
>>>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopologyWithOpts(nimbus.clj:953)
>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>> at
>>>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopology(nimbus.clj:966)
>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>> at
>>>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1240)
>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>> at
>>>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1228)
>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>> at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32)
>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>> at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34)
>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>> at
>>>> org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632)
>>>> ~[storm-core-0.9.4.jar:0.9.4]
>>>> at
>>>> org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201)
>>>> [storm-core-0.9.4.jar:0.9.4]
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> [na:1.7.0_45]
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> [na:1.7.0_45]
>>>> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, May 7, 2015 at 5:36 PM, Nishu <ni...@gmail.com> wrote:
>>>>
>>>>> You can check if there are free workers available to run the topology
>>>>> or not.
>>>>>
>>>>> Thanks,
>>>>> Nishu Tayal
>>>>> On 7 May 2015 17:08, "bhargav sarvepalli" <bo...@gmail.com> wrote:
>>>>>
>>>>>> Your nimbus has gone down, probably because there is no space left in
>>>>>> your machine.
>>>>>>
>>>>>> On Thu, May 7, 2015 at 3:25 PM, aarthi ranganathan <
>>>>>> aarthirangan@gmail.com> wrote:
>>>>>>
>>>>>>> The zookeeper exists in the same machine along with storm. So I
>>>>>>> guess the ports are accessible.
>>>>>>>
>>>>>>> On Wed, May 6, 2015 at 6:44 PM, Aditya Rajan <
>>>>>>> aditya.rajan@whizdm.com> wrote:
>>>>>>>
>>>>>>>> I would assume it's an issue with zookeeper. Have you checked if
>>>>>>>> the relevant ports are open?
>>>>>>>> On May 6, 2015 6:02 PM, "aarthi ranganathan" <
>>>>>>>> aarthirangan@gmail.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> We are using Storm Submitter via tomcat to submit topology to
>>>>>>>>> storm cluster. The jar does not get uploaded to storm in the same machine
>>>>>>>>> as well as a different machine. We are using apache-storm -0.9.4.This error
>>>>>>>>> occurs intermittently. Following is the stack trace
>>>>>>>>>
>>>>>>>>> java.lang.RuntimeException:
>>>>>>>>> org.apache.thrift7.transport.TTransportException
>>>>>>>>> at
>>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:141)
>>>>>>>>> at
>>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:70)
>>>>>>>>> at
>>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.submitTopology(StormProcessor.java:148)
>>>>>>>>> at
>>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.access$000(StormProcessor.java:48)
>>>>>>>>> at
>>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor$1.call(StormProcessor.java:111)
>>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>>>> ... 3 more
>>>>>>>>> Caused by: org.apache.thrift7.transport.TTransportException
>>>>>>>>> at
>>>>>>>>> org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>>>>>>>>> at
>>>>>>>>> org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
>>>>>>>>> at
>>>>>>>>> org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
>>>>>>>>> at
>>>>>>>>> org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101)
>>>>>>>>> at
>>>>>>>>> org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
>>>>>>>>> at
>>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
>>>>>>>>> at
>>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
>>>>>>>>> at
>>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
>>>>>>>>> at
>>>>>>>>> org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:69)
>>>>>>>>> at
>>>>>>>>> backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:162)
>>>>>>>>> at
>>>>>>>>> backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:146)
>>>>>>>>> at
>>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:127)
>>>>>>>>> ... 8 more
>>>>>>>>>
>>>>>>>>> We are not able to reproduce the issue for a particular scenario.
>>>>>>>>> It occurs randomly. Can u throw some light on the issue? We have checked
>>>>>>>>> the firewall settings and there is nothing blocked.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Aarthi R
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Aarthi R
>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>> --
>>>> Aarthi R
>>>>
>>>
>>>
>>
>
>
> --
> Aarthi R
>
Re: Storm jar transport error
Posted by aarthi ranganathan <aa...@gmail.com>.
I tried clearing the local state and also restarted storm. Still getting
Illegal Argument Exception. Since jar is not getting uploaded due to
illegal argument exception, the topology does not exist in the zookeeper.
The jar is getting uploaded from tomcat to nimbus inbox. Its not getting
copied to storm dist folder. Im posting the recent stack trace where look
into highlighted parts in bold. The jar uploaded starts with <stormjar-e7>
while the jar that is trying to get transferred is <stormjar-f8>. Is storm
caching old jars and trying to run? Our team also saw this storm issue
https://issues.apache.org/jira/browse/STORM-467 which is still unresolved
where they mention trying to execute non-existent jars as the issue.
2015-05-08T11:58:36.122+0530 b.s.d.nimbus [INFO] Uploading file from client
to
/home/xbdev/storm/nimbus/inbox/stormjar-e7c13d9a-3aed-4232-9d20-6813e9c45c03.jar
2015-05-08T11:58:38.546+0530 b.s.d.nimbus [INFO] Finished uploading file
from client:
*/home/xbdev/storm/nimbus/inbox/stormjar-e7c13d9a-3aed-4232-9d20-6813e9c45c03.jar*
2015-05-08T11:58:38.710+0530 b.s.d.nimbus [INFO] Received topology
submission for topology-1431056498295 with conf {"storm.id"
"topology-1431056498295-1-1431066518", "cassandra-config" {"cassandra.host"
"10.237.247.13", "cassandra.keyspace" ["stormks"]}, "nimbus.host"
"10.237.247.3", "storm.zookeeper.port" 2181, "storm.zookeeper.servers"
["10.237.247.3"], "topology.acker.executors" nil,
"topology.kryo.decorators" (), "topology.name" "topology-1431056498295",
"topology.debug" true, "topology.kryo.register" nil,
"storm.elasticsearch.hosts" "10.237.247.13:9300", "
storm.elasticsearch.cluster.name" "HAP_Development_1.0", "topology.workers"
2, "nimbus.thrift.port" 6627, "storm.thrift.transport"
"backtype.storm.security.auth.SimpleTransportPlugin",
"topology.max.task.parallelism" nil}
2015-05-08T11:58:38.716+0530 b.s.d.nimbus [WARN] Topology submission
exception. (topology name='topology-1431056498295')
#<IllegalArgumentException java.lang.IllegalArgumentException:
*/home/xbdev/storm/nimbus/inbox/stormjar-f8982d19-81d0-420a-a847-7243eb6d4181.jar*
to copy to
/home/xbdev/storm/nimbus/stormdist/topology-1431056498295-1-1431066518 does
not exist!>
2015-05-08T11:58:38.722+0530 o.a.t.s.TNonblockingServer [ERROR] Unexpected
exception while invoking!
java.lang.IllegalArgumentException:
/home/xbdev/storm/nimbus/inbox/stormjar-f8982d19-81d0-420a-a847-7243eb6d4181.jar
to copy to
/home/xbdev/storm/nimbus/stormdist/topology-1431056498295-1-1431066518 does
not exist!
at backtype.storm.daemon.nimbus$fn__4393.invoke(nimbus.clj:1173)
~[storm-core-0.9.4.jar:0.9.4]
at clojure.lang.MultiFn.invoke(MultiFn.java:236) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:307)
~[storm-core-0.9.4.jar:0.9.4]
at
backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopologyWithOpts(nimbus.clj:953)
~[storm-core-0.9.4.jar:0.9.4]
at
backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopology(nimbus.clj:966)
~[storm-core-0.9.4.jar:0.9.4]
at
backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1240)
~[storm-core-0.9.4.jar:0.9.4]
at
backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1228)
~[storm-core-0.9.4.jar:0.9.4]
at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32)
~[storm-core-0.9.4.jar:0.9.4]
at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34)
~[storm-core-0.9.4.jar:0.9.4]
at
org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632)
~[storm-core-0.9.4.jar:0.9.4]
at
org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201)
[storm-core-0.9.4.jar:0.9.4]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
On Fri, May 8, 2015 at 11:49 AM, Erik Weathers <ew...@groupon.com>
wrote:
> If that doesn't work, then you might need to remove the state in zookeeper
> related to this topology.
> i.e.,
>
> - /storm/storms/<topology-id>
> - /storm/assignments/<topology-id>
>
> - Erik
>
> On Thu, May 7, 2015 at 11:18 PM, bhargav sarvepalli <bo...@gmail.com>
> wrote:
>
>> Try deleting your local state folder in supervisors and storm nimbus data
>> folder, looks like its trying to find topology which does not exist.
>>
>> On Fri, May 8, 2015 at 11:08 AM, aarthi ranganathan <
>> aarthirangan@gmail.com> wrote:
>>
>>> Hi
>>>
>>> While checking nimbus log, Im getting the following stack track trace..
>>> The previous stack trace was while running from tomcat application , which
>>> generates an illegal argument exception in nimbus logs
>>>
>>> 2015-05-06T14:27:57.718+0530 o.a.t.s.TNonblockingServer [ERROR]
>>> Unexpected exception while invoking!
>>> java.lang.IllegalArgumentException:
>>> /home/xbdev/storm/nimbus/inbox/stormjar-e9973a4f-7dc2-4cd5-a1ef-4e8c95ec9a34.jar
>>> to copy to
>>> /home/xbdev/storm/nimbus/stormdist/topology-1430892623717-3-1430902677 does
>>> not exist!
>>> at backtype.storm.daemon.nimbus$fn__4393.invoke(nimbus.clj:1173)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>> at clojure.lang.MultiFn.invoke(MultiFn.java:236) ~[clojure-1.5.1.jar:na]
>>> at backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:307)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>> at
>>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopologyWithOpts(nimbus.clj:953)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>> at
>>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopology(nimbus.clj:966)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>> at
>>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1240)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>> at
>>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1228)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>> at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>> at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>> at
>>> org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632)
>>> ~[storm-core-0.9.4.jar:0.9.4]
>>> at
>>> org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201)
>>> [storm-core-0.9.4.jar:0.9.4]
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> [na:1.7.0_45]
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> [na:1.7.0_45]
>>> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
>>>
>>>
>>>
>>>
>>> On Thu, May 7, 2015 at 5:36 PM, Nishu <ni...@gmail.com> wrote:
>>>
>>>> You can check if there are free workers available to run the topology
>>>> or not.
>>>>
>>>> Thanks,
>>>> Nishu Tayal
>>>> On 7 May 2015 17:08, "bhargav sarvepalli" <bo...@gmail.com> wrote:
>>>>
>>>>> Your nimbus has gone down, probably because there is no space left in
>>>>> your machine.
>>>>>
>>>>> On Thu, May 7, 2015 at 3:25 PM, aarthi ranganathan <
>>>>> aarthirangan@gmail.com> wrote:
>>>>>
>>>>>> The zookeeper exists in the same machine along with storm. So I guess
>>>>>> the ports are accessible.
>>>>>>
>>>>>> On Wed, May 6, 2015 at 6:44 PM, Aditya Rajan <aditya.rajan@whizdm.com
>>>>>> > wrote:
>>>>>>
>>>>>>> I would assume it's an issue with zookeeper. Have you checked if the
>>>>>>> relevant ports are open?
>>>>>>> On May 6, 2015 6:02 PM, "aarthi ranganathan" <aa...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> We are using Storm Submitter via tomcat to submit topology to storm
>>>>>>>> cluster. The jar does not get uploaded to storm in the same machine as well
>>>>>>>> as a different machine. We are using apache-storm -0.9.4.This error occurs
>>>>>>>> intermittently. Following is the stack trace
>>>>>>>>
>>>>>>>> java.lang.RuntimeException:
>>>>>>>> org.apache.thrift7.transport.TTransportException
>>>>>>>> at
>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:141)
>>>>>>>> at
>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:70)
>>>>>>>> at
>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.submitTopology(StormProcessor.java:148)
>>>>>>>> at
>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.access$000(StormProcessor.java:48)
>>>>>>>> at
>>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor$1.call(StormProcessor.java:111)
>>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>>> ... 3 more
>>>>>>>> Caused by: org.apache.thrift7.transport.TTransportException
>>>>>>>> at
>>>>>>>> org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>>>>>>>> at
>>>>>>>> org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
>>>>>>>> at
>>>>>>>> org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
>>>>>>>> at
>>>>>>>> org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101)
>>>>>>>> at
>>>>>>>> org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
>>>>>>>> at
>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
>>>>>>>> at
>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
>>>>>>>> at
>>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
>>>>>>>> at
>>>>>>>> org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:69)
>>>>>>>> at
>>>>>>>> backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:162)
>>>>>>>> at
>>>>>>>> backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:146)
>>>>>>>> at
>>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:127)
>>>>>>>> ... 8 more
>>>>>>>>
>>>>>>>> We are not able to reproduce the issue for a particular scenario.
>>>>>>>> It occurs randomly. Can u throw some light on the issue? We have checked
>>>>>>>> the firewall settings and there is nothing blocked.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Aarthi R
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Aarthi R
>>>>>>
>>>>>
>>>>>
>>>
>>>
>>> --
>>> Aarthi R
>>>
>>
>>
>
--
Aarthi R
Re: Storm jar transport error
Posted by Erik Weathers <ew...@groupon.com>.
If that doesn't work, then you might need to remove the state in zookeeper
related to this topology.
i.e.,
- /storm/storms/<topology-id>
- /storm/assignments/<topology-id>
- Erik
On Thu, May 7, 2015 at 11:18 PM, bhargav sarvepalli <bo...@gmail.com>
wrote:
> Try deleting your local state folder in supervisors and storm nimbus data
> folder, looks like its trying to find topology which does not exist.
>
> On Fri, May 8, 2015 at 11:08 AM, aarthi ranganathan <
> aarthirangan@gmail.com> wrote:
>
>> Hi
>>
>> While checking nimbus log, Im getting the following stack track trace..
>> The previous stack trace was while running from tomcat application , which
>> generates an illegal argument exception in nimbus logs
>>
>> 2015-05-06T14:27:57.718+0530 o.a.t.s.TNonblockingServer [ERROR]
>> Unexpected exception while invoking!
>> java.lang.IllegalArgumentException:
>> /home/xbdev/storm/nimbus/inbox/stormjar-e9973a4f-7dc2-4cd5-a1ef-4e8c95ec9a34.jar
>> to copy to
>> /home/xbdev/storm/nimbus/stormdist/topology-1430892623717-3-1430902677 does
>> not exist!
>> at backtype.storm.daemon.nimbus$fn__4393.invoke(nimbus.clj:1173)
>> ~[storm-core-0.9.4.jar:0.9.4]
>> at clojure.lang.MultiFn.invoke(MultiFn.java:236) ~[clojure-1.5.1.jar:na]
>> at backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:307)
>> ~[storm-core-0.9.4.jar:0.9.4]
>> at
>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopologyWithOpts(nimbus.clj:953)
>> ~[storm-core-0.9.4.jar:0.9.4]
>> at
>> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopology(nimbus.clj:966)
>> ~[storm-core-0.9.4.jar:0.9.4]
>> at
>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1240)
>> ~[storm-core-0.9.4.jar:0.9.4]
>> at
>> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1228)
>> ~[storm-core-0.9.4.jar:0.9.4]
>> at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32)
>> ~[storm-core-0.9.4.jar:0.9.4]
>> at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34)
>> ~[storm-core-0.9.4.jar:0.9.4]
>> at
>> org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632)
>> ~[storm-core-0.9.4.jar:0.9.4]
>> at
>> org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201)
>> [storm-core-0.9.4.jar:0.9.4]
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> [na:1.7.0_45]
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> [na:1.7.0_45]
>> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
>>
>>
>>
>>
>> On Thu, May 7, 2015 at 5:36 PM, Nishu <ni...@gmail.com> wrote:
>>
>>> You can check if there are free workers available to run the topology or
>>> not.
>>>
>>> Thanks,
>>> Nishu Tayal
>>> On 7 May 2015 17:08, "bhargav sarvepalli" <bo...@gmail.com> wrote:
>>>
>>>> Your nimbus has gone down, probably because there is no space left in
>>>> your machine.
>>>>
>>>> On Thu, May 7, 2015 at 3:25 PM, aarthi ranganathan <
>>>> aarthirangan@gmail.com> wrote:
>>>>
>>>>> The zookeeper exists in the same machine along with storm. So I guess
>>>>> the ports are accessible.
>>>>>
>>>>> On Wed, May 6, 2015 at 6:44 PM, Aditya Rajan <ad...@whizdm.com>
>>>>> wrote:
>>>>>
>>>>>> I would assume it's an issue with zookeeper. Have you checked if the
>>>>>> relevant ports are open?
>>>>>> On May 6, 2015 6:02 PM, "aarthi ranganathan" <aa...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> We are using Storm Submitter via tomcat to submit topology to storm
>>>>>>> cluster. The jar does not get uploaded to storm in the same machine as well
>>>>>>> as a different machine. We are using apache-storm -0.9.4.This error occurs
>>>>>>> intermittently. Following is the stack trace
>>>>>>>
>>>>>>> java.lang.RuntimeException:
>>>>>>> org.apache.thrift7.transport.TTransportException
>>>>>>> at
>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:141)
>>>>>>> at
>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:70)
>>>>>>> at
>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.submitTopology(StormProcessor.java:148)
>>>>>>> at
>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.access$000(StormProcessor.java:48)
>>>>>>> at
>>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor$1.call(StormProcessor.java:111)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> ... 3 more
>>>>>>> Caused by: org.apache.thrift7.transport.TTransportException
>>>>>>> at
>>>>>>> org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>>>>>>> at
>>>>>>> org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
>>>>>>> at
>>>>>>> org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
>>>>>>> at
>>>>>>> org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101)
>>>>>>> at
>>>>>>> org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
>>>>>>> at
>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
>>>>>>> at
>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
>>>>>>> at
>>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
>>>>>>> at
>>>>>>> org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:69)
>>>>>>> at
>>>>>>> backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:162)
>>>>>>> at
>>>>>>> backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:146)
>>>>>>> at
>>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:127)
>>>>>>> ... 8 more
>>>>>>>
>>>>>>> We are not able to reproduce the issue for a particular scenario. It
>>>>>>> occurs randomly. Can u throw some light on the issue? We have checked the
>>>>>>> firewall settings and there is nothing blocked.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Aarthi R
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Aarthi R
>>>>>
>>>>
>>>>
>>
>>
>> --
>> Aarthi R
>>
>
>
Re: Storm jar transport error
Posted by bhargav sarvepalli <bo...@gmail.com>.
Try deleting your local state folder in supervisors and storm nimbus data
folder, looks like its trying to find topology which does not exist.
On Fri, May 8, 2015 at 11:08 AM, aarthi ranganathan <aa...@gmail.com>
wrote:
> Hi
>
> While checking nimbus log, Im getting the following stack track trace..
> The previous stack trace was while running from tomcat application , which
> generates an illegal argument exception in nimbus logs
>
> 2015-05-06T14:27:57.718+0530 o.a.t.s.TNonblockingServer [ERROR] Unexpected
> exception while invoking!
> java.lang.IllegalArgumentException:
> /home/xbdev/storm/nimbus/inbox/stormjar-e9973a4f-7dc2-4cd5-a1ef-4e8c95ec9a34.jar
> to copy to
> /home/xbdev/storm/nimbus/stormdist/topology-1430892623717-3-1430902677 does
> not exist!
> at backtype.storm.daemon.nimbus$fn__4393.invoke(nimbus.clj:1173)
> ~[storm-core-0.9.4.jar:0.9.4]
> at clojure.lang.MultiFn.invoke(MultiFn.java:236) ~[clojure-1.5.1.jar:na]
> at backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:307)
> ~[storm-core-0.9.4.jar:0.9.4]
> at
> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopologyWithOpts(nimbus.clj:953)
> ~[storm-core-0.9.4.jar:0.9.4]
> at
> backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopology(nimbus.clj:966)
> ~[storm-core-0.9.4.jar:0.9.4]
> at
> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1240)
> ~[storm-core-0.9.4.jar:0.9.4]
> at
> backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1228)
> ~[storm-core-0.9.4.jar:0.9.4]
> at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32)
> ~[storm-core-0.9.4.jar:0.9.4]
> at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34)
> ~[storm-core-0.9.4.jar:0.9.4]
> at
> org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632)
> ~[storm-core-0.9.4.jar:0.9.4]
> at
> org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201)
> [storm-core-0.9.4.jar:0.9.4]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
>
>
>
>
> On Thu, May 7, 2015 at 5:36 PM, Nishu <ni...@gmail.com> wrote:
>
>> You can check if there are free workers available to run the topology or
>> not.
>>
>> Thanks,
>> Nishu Tayal
>> On 7 May 2015 17:08, "bhargav sarvepalli" <bo...@gmail.com> wrote:
>>
>>> Your nimbus has gone down, probably because there is no space left in
>>> your machine.
>>>
>>> On Thu, May 7, 2015 at 3:25 PM, aarthi ranganathan <
>>> aarthirangan@gmail.com> wrote:
>>>
>>>> The zookeeper exists in the same machine along with storm. So I guess
>>>> the ports are accessible.
>>>>
>>>> On Wed, May 6, 2015 at 6:44 PM, Aditya Rajan <ad...@whizdm.com>
>>>> wrote:
>>>>
>>>>> I would assume it's an issue with zookeeper. Have you checked if the
>>>>> relevant ports are open?
>>>>> On May 6, 2015 6:02 PM, "aarthi ranganathan" <aa...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> We are using Storm Submitter via tomcat to submit topology to storm
>>>>>> cluster. The jar does not get uploaded to storm in the same machine as well
>>>>>> as a different machine. We are using apache-storm -0.9.4.This error occurs
>>>>>> intermittently. Following is the stack trace
>>>>>>
>>>>>> java.lang.RuntimeException:
>>>>>> org.apache.thrift7.transport.TTransportException
>>>>>> at
>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:141)
>>>>>> at
>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:70)
>>>>>> at
>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.submitTopology(StormProcessor.java:148)
>>>>>> at
>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.access$000(StormProcessor.java:48)
>>>>>> at
>>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor$1.call(StormProcessor.java:111)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> ... 3 more
>>>>>> Caused by: org.apache.thrift7.transport.TTransportException
>>>>>> at
>>>>>> org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>>>>>> at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
>>>>>> at
>>>>>> org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
>>>>>> at
>>>>>> org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101)
>>>>>> at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
>>>>>> at
>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
>>>>>> at
>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
>>>>>> at
>>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
>>>>>> at
>>>>>> org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:69)
>>>>>> at
>>>>>> backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:162)
>>>>>> at
>>>>>> backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:146)
>>>>>> at
>>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:127)
>>>>>> ... 8 more
>>>>>>
>>>>>> We are not able to reproduce the issue for a particular scenario. It
>>>>>> occurs randomly. Can u throw some light on the issue? We have checked the
>>>>>> firewall settings and there is nothing blocked.
>>>>>>
>>>>>> Thanks
>>>>>> Aarthi R
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Aarthi R
>>>>
>>>
>>>
>
>
> --
> Aarthi R
>
Re: Storm jar transport error
Posted by aarthi ranganathan <aa...@gmail.com>.
Hi
While checking nimbus log, Im getting the following stack track trace.. The
previous stack trace was while running from tomcat application , which
generates an illegal argument exception in nimbus logs
2015-05-06T14:27:57.718+0530 o.a.t.s.TNonblockingServer [ERROR] Unexpected
exception while invoking!
java.lang.IllegalArgumentException:
/home/xbdev/storm/nimbus/inbox/stormjar-e9973a4f-7dc2-4cd5-a1ef-4e8c95ec9a34.jar
to copy to
/home/xbdev/storm/nimbus/stormdist/topology-1430892623717-3-1430902677 does
not exist!
at backtype.storm.daemon.nimbus$fn__4393.invoke(nimbus.clj:1173)
~[storm-core-0.9.4.jar:0.9.4]
at clojure.lang.MultiFn.invoke(MultiFn.java:236) ~[clojure-1.5.1.jar:na]
at backtype.storm.daemon.nimbus$setup_storm_code.invoke(nimbus.clj:307)
~[storm-core-0.9.4.jar:0.9.4]
at
backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopologyWithOpts(nimbus.clj:953)
~[storm-core-0.9.4.jar:0.9.4]
at
backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto__$reify__4303.submitTopology(nimbus.clj:966)
~[storm-core-0.9.4.jar:0.9.4]
at
backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1240)
~[storm-core-0.9.4.jar:0.9.4]
at
backtype.storm.generated.Nimbus$Processor$submitTopology.getResult(Nimbus.java:1228)
~[storm-core-0.9.4.jar:0.9.4]
at org.apache.thrift7.ProcessFunction.process(ProcessFunction.java:32)
~[storm-core-0.9.4.jar:0.9.4]
at org.apache.thrift7.TBaseProcessor.process(TBaseProcessor.java:34)
~[storm-core-0.9.4.jar:0.9.4]
at
org.apache.thrift7.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:632)
~[storm-core-0.9.4.jar:0.9.4]
at
org.apache.thrift7.server.THsHaServer$Invocation.run(THsHaServer.java:201)
[storm-core-0.9.4.jar:0.9.4]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
On Thu, May 7, 2015 at 5:36 PM, Nishu <ni...@gmail.com> wrote:
> You can check if there are free workers available to run the topology or
> not.
>
> Thanks,
> Nishu Tayal
> On 7 May 2015 17:08, "bhargav sarvepalli" <bo...@gmail.com> wrote:
>
>> Your nimbus has gone down, probably because there is no space left in
>> your machine.
>>
>> On Thu, May 7, 2015 at 3:25 PM, aarthi ranganathan <
>> aarthirangan@gmail.com> wrote:
>>
>>> The zookeeper exists in the same machine along with storm. So I guess
>>> the ports are accessible.
>>>
>>> On Wed, May 6, 2015 at 6:44 PM, Aditya Rajan <ad...@whizdm.com>
>>> wrote:
>>>
>>>> I would assume it's an issue with zookeeper. Have you checked if the
>>>> relevant ports are open?
>>>> On May 6, 2015 6:02 PM, "aarthi ranganathan" <aa...@gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>> Hi
>>>>>
>>>>> We are using Storm Submitter via tomcat to submit topology to storm
>>>>> cluster. The jar does not get uploaded to storm in the same machine as well
>>>>> as a different machine. We are using apache-storm -0.9.4.This error occurs
>>>>> intermittently. Following is the stack trace
>>>>>
>>>>> java.lang.RuntimeException:
>>>>> org.apache.thrift7.transport.TTransportException
>>>>> at
>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:141)
>>>>> at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:70)
>>>>> at
>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.submitTopology(StormProcessor.java:148)
>>>>> at
>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.access$000(StormProcessor.java:48)
>>>>> at
>>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor$1.call(StormProcessor.java:111)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>> ... 3 more
>>>>> Caused by: org.apache.thrift7.transport.TTransportException
>>>>> at
>>>>> org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>>>>> at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
>>>>> at
>>>>> org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
>>>>> at
>>>>> org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101)
>>>>> at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
>>>>> at
>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
>>>>> at
>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
>>>>> at
>>>>> org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
>>>>> at
>>>>> org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:69)
>>>>> at
>>>>> backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:162)
>>>>> at
>>>>> backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:146)
>>>>> at
>>>>> backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:127)
>>>>> ... 8 more
>>>>>
>>>>> We are not able to reproduce the issue for a particular scenario. It
>>>>> occurs randomly. Can u throw some light on the issue? We have checked the
>>>>> firewall settings and there is nothing blocked.
>>>>>
>>>>> Thanks
>>>>> Aarthi R
>>>>>
>>>>
>>>
>>>
>>> --
>>> Aarthi R
>>>
>>
>>
--
Aarthi R
Re: Storm jar transport error
Posted by Nishu <ni...@gmail.com>.
You can check if there are free workers available to run the topology or
not.
Thanks,
Nishu Tayal
On 7 May 2015 17:08, "bhargav sarvepalli" <bo...@gmail.com> wrote:
> Your nimbus has gone down, probably because there is no space left in your
> machine.
>
> On Thu, May 7, 2015 at 3:25 PM, aarthi ranganathan <aarthirangan@gmail.com
> > wrote:
>
>> The zookeeper exists in the same machine along with storm. So I guess the
>> ports are accessible.
>>
>> On Wed, May 6, 2015 at 6:44 PM, Aditya Rajan <ad...@whizdm.com>
>> wrote:
>>
>>> I would assume it's an issue with zookeeper. Have you checked if the
>>> relevant ports are open?
>>> On May 6, 2015 6:02 PM, "aarthi ranganathan" <aa...@gmail.com>
>>> wrote:
>>>
>>>>
>>>> Hi
>>>>
>>>> We are using Storm Submitter via tomcat to submit topology to storm
>>>> cluster. The jar does not get uploaded to storm in the same machine as well
>>>> as a different machine. We are using apache-storm -0.9.4.This error occurs
>>>> intermittently. Following is the stack trace
>>>>
>>>> java.lang.RuntimeException:
>>>> org.apache.thrift7.transport.TTransportException
>>>> at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:141)
>>>> at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:70)
>>>> at
>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.submitTopology(StormProcessor.java:148)
>>>> at
>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.access$000(StormProcessor.java:48)
>>>> at
>>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor$1.call(StormProcessor.java:111)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> ... 3 more
>>>> Caused by: org.apache.thrift7.transport.TTransportException
>>>> at
>>>> org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>>>> at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
>>>> at
>>>> org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
>>>> at
>>>> org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101)
>>>> at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
>>>> at
>>>> org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
>>>> at
>>>> org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
>>>> at
>>>> org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
>>>> at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:69)
>>>> at
>>>> backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:162)
>>>> at
>>>> backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:146)
>>>> at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:127)
>>>> ... 8 more
>>>>
>>>> We are not able to reproduce the issue for a particular scenario. It
>>>> occurs randomly. Can u throw some light on the issue? We have checked the
>>>> firewall settings and there is nothing blocked.
>>>>
>>>> Thanks
>>>> Aarthi R
>>>>
>>>
>>
>>
>> --
>> Aarthi R
>>
>
>
Re: Storm jar transport error
Posted by bhargav sarvepalli <bo...@gmail.com>.
Your nimbus has gone down, probably because there is no space left in your
machine.
On Thu, May 7, 2015 at 3:25 PM, aarthi ranganathan <aa...@gmail.com>
wrote:
> The zookeeper exists in the same machine along with storm. So I guess the
> ports are accessible.
>
> On Wed, May 6, 2015 at 6:44 PM, Aditya Rajan <ad...@whizdm.com>
> wrote:
>
>> I would assume it's an issue with zookeeper. Have you checked if the
>> relevant ports are open?
>> On May 6, 2015 6:02 PM, "aarthi ranganathan" <aa...@gmail.com>
>> wrote:
>>
>>>
>>> Hi
>>>
>>> We are using Storm Submitter via tomcat to submit topology to storm
>>> cluster. The jar does not get uploaded to storm in the same machine as well
>>> as a different machine. We are using apache-storm -0.9.4.This error occurs
>>> intermittently. Following is the stack trace
>>>
>>> java.lang.RuntimeException:
>>> org.apache.thrift7.transport.TTransportException
>>> at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:141)
>>> at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:70)
>>> at
>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.submitTopology(StormProcessor.java:148)
>>> at
>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.access$000(StormProcessor.java:48)
>>> at
>>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor$1.call(StormProcessor.java:111)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>> ... 3 more
>>> Caused by: org.apache.thrift7.transport.TTransportException
>>> at
>>> org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>>> at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
>>> at
>>> org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
>>> at
>>> org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101)
>>> at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
>>> at
>>> org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
>>> at
>>> org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
>>> at
>>> org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
>>> at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:69)
>>> at
>>> backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:162)
>>> at backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:146)
>>> at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:127)
>>> ... 8 more
>>>
>>> We are not able to reproduce the issue for a particular scenario. It
>>> occurs randomly. Can u throw some light on the issue? We have checked the
>>> firewall settings and there is nothing blocked.
>>>
>>> Thanks
>>> Aarthi R
>>>
>>
>
>
> --
> Aarthi R
>
Re: Storm jar transport error
Posted by aarthi ranganathan <aa...@gmail.com>.
The zookeeper exists in the same machine along with storm. So I guess the
ports are accessible.
On Wed, May 6, 2015 at 6:44 PM, Aditya Rajan <ad...@whizdm.com>
wrote:
> I would assume it's an issue with zookeeper. Have you checked if the
> relevant ports are open?
> On May 6, 2015 6:02 PM, "aarthi ranganathan" <aa...@gmail.com>
> wrote:
>
>>
>> Hi
>>
>> We are using Storm Submitter via tomcat to submit topology to storm
>> cluster. The jar does not get uploaded to storm in the same machine as well
>> as a different machine. We are using apache-storm -0.9.4.This error occurs
>> intermittently. Following is the stack trace
>>
>> java.lang.RuntimeException:
>> org.apache.thrift7.transport.TTransportException
>> at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:141)
>> at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:70)
>> at
>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.submitTopology(StormProcessor.java:148)
>> at
>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.access$000(StormProcessor.java:48)
>> at
>> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor$1.call(StormProcessor.java:111)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> ... 3 more
>> Caused by: org.apache.thrift7.transport.TTransportException
>> at
>> org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>> at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
>> at
>> org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
>> at
>> org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101)
>> at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
>> at
>> org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
>> at
>> org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
>> at
>> org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
>> at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:69)
>> at
>> backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:162)
>> at backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:146)
>> at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:127)
>> ... 8 more
>>
>> We are not able to reproduce the issue for a particular scenario. It
>> occurs randomly. Can u throw some light on the issue? We have checked the
>> firewall settings and there is nothing blocked.
>>
>> Thanks
>> Aarthi R
>>
>
--
Aarthi R
Re: Storm jar transport error
Posted by Aditya Rajan <ad...@whizdm.com>.
I would assume it's an issue with zookeeper. Have you checked if the
relevant ports are open?
On May 6, 2015 6:02 PM, "aarthi ranganathan" <aa...@gmail.com> wrote:
>
> Hi
>
> We are using Storm Submitter via tomcat to submit topology to storm
> cluster. The jar does not get uploaded to storm in the same machine as well
> as a different machine. We are using apache-storm -0.9.4.This error occurs
> intermittently. Following is the stack trace
>
> java.lang.RuntimeException:
> org.apache.thrift7.transport.TTransportException
> at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:141)
> at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:70)
> at
> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.submitTopology(StormProcessor.java:148)
> at
> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor.access$000(StormProcessor.java:48)
> at
> com.cognizant.hpc.xstreambatch.processor.stream.storm.StormProcessor$1.call(StormProcessor.java:111)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ... 3 more
> Caused by: org.apache.thrift7.transport.TTransportException
> at
> org.apache.thrift7.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
> at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift7.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
> at
> org.apache.thrift7.transport.TFramedTransport.read(TFramedTransport.java:101)
> at org.apache.thrift7.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift7.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
> at
> org.apache.thrift7.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
> at
> org.apache.thrift7.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
> at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:69)
> at
> backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:162)
> at backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:146)
> at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:127)
> ... 8 more
>
> We are not able to reproduce the issue for a particular scenario. It
> occurs randomly. Can u throw some light on the issue? We have checked the
> firewall settings and there is nothing blocked.
>
> Thanks
> Aarthi R
>