You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Vikas Agarwal <vi...@infoobjects.com> on 2014/08/25 13:55:04 UTC
Storm not processing topology without logs
Hi,
I have started to explore the Storm for distributed processing for our use
case which we were earlier fulfilling by JMS based MQ system. Topology
worked after some efforts. It has one spout (KafkaSpout from kafka-storm
project) and 3 bolts. First bolt sets context for other two bolts which in
turn do some processing on the tuples and persist the analyzed results in
some DB (Mongo, Solr, HBase etc).
Recently the topology stopped working. I am able to submit the topology and
it does not throw any error in submitting the topology, however, nimbus.log
or worker-6701.log files are not showing any progress and eventually
topology does not consume any message. I don't have doubt on KafkaSpout
because if it was the culprit, at least some initialization logs of spout
and bolts should have been there in nimbus.log or worker-xxxx.log. Isn't it?
Here is the snippet of nimbus.log after uploading the jar to cluster
Uploading file from client to
/hadoop/storm/nimbus/inbox/stormjar-31fe068b-337b-428f-8ae2-fe13c706b2ab.jar
2014-08-25 07:07:49 b.s.d.nimbus [INFO] Finished uploading file from
client:
/hadoop/storm/nimbus/inbox/stormjar-31fe068b-337b-428f-8ae2-fe13c706b2ab.jar
2014-08-25 07:07:49 b.s.d.nimbus [INFO] Received topology submission for
aleads with conf {"topology.max.task.parallelism" nil,
"topology.acker.executors" nil, "topology.kryo.register" nil,
"topology.kryo.decorators" (), "topology.name" "aleads", "storm.id"
"aleads-3-1408964869", "modelId" "ut", "topology.workers" 1,
"topology.debug" true}
2014-08-25 07:07:50 b.s.d.nimbus [INFO] Activating aleads:
aleads-3-1408964869
2014-08-25 07:07:50 b.s.s.EvenScheduler [INFO] Available slots:
(["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701]
["e56c2cc7-d35a-4355-9906-506618ff70c5" 6700])
2014-08-25 07:07:50 b.s.d.nimbus [INFO] Setting new assignment for topology
id aleads-3-1408964869:
#backtype.storm.daemon.common.Assignment{:master-code-dir
"/hadoop/storm/nimbus/stormdist/aleads-3-1408964869", :node->host
{"e56c2cc7-d35a-4355-9906-506618ff70c5" "hdp.ambari"}, :executor->node+port
{[2 2] ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [3 3]
["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [4 4]
["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [5 5]
["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [6 6]
["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [7 7]
["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [8 8]
["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [9 9]
["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [1 1]
["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701]}, :executor->start-time-secs
{[1 1] 1408964870, [9 9] 1408964870, [8 8] 1408964870, [7 7] 1408964870, [6
6] 1408964870, [5 5] 1408964870, [4 4] 1408964870, [3 3] 1408964870, [2 2]
1408964870}}
Can anyone guess what I have done wrong and why Storm is not giving any
error log anywhere.
Storm version is 0.9.1.2.1.3.0-563 (Installed via HortonWorks)
Kafka version is 2.10-0.8.1.1
Storm-Kafka version 0.9.2-incubating
--
Regards,
Vikas Agarwal
91 – 9928301411
InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax
Re: Storm not processing topology without logs
Posted by Vikas Agarwal <vi...@infoobjects.com>.
Found the fix. I was stuck in this problem from last 3-4 days and it was
waiting just for joining the storm mailing list to be resolved. :)
I found something in supervisor.log this time, it was dumping <jar_UUID>
still hasn't started with the actual worker java command which is failing.
However, it was not showing any error. So, I copied the command from the
logs and directly run on console and it showed me the root cause. Somehow,
localhost was get appended to hdp.ambari (which was my host name) and due
to it was not able to find the server to run the command on. :(
On Mon, Aug 25, 2014 at 5:25 PM, Vikas Agarwal <vi...@infoobjects.com>
wrote:
> Hi,
>
> I have started to explore the Storm for distributed processing for our use
> case which we were earlier fulfilling by JMS based MQ system. Topology
> worked after some efforts. It has one spout (KafkaSpout from kafka-storm
> project) and 3 bolts. First bolt sets context for other two bolts which in
> turn do some processing on the tuples and persist the analyzed results in
> some DB (Mongo, Solr, HBase etc).
>
> Recently the topology stopped working. I am able to submit the topology
> and it does not throw any error in submitting the topology, however,
> nimbus.log or worker-6701.log files are not showing any progress and
> eventually topology does not consume any message. I don't have doubt on
> KafkaSpout because if it was the culprit, at least some initialization logs
> of spout and bolts should have been there in nimbus.log or worker-xxxx.log.
> Isn't it?
>
> Here is the snippet of nimbus.log after uploading the jar to cluster
>
> Uploading file from client to
> /hadoop/storm/nimbus/inbox/stormjar-31fe068b-337b-428f-8ae2-fe13c706b2ab.jar
> 2014-08-25 07:07:49 b.s.d.nimbus [INFO] Finished uploading file from
> client:
> /hadoop/storm/nimbus/inbox/stormjar-31fe068b-337b-428f-8ae2-fe13c706b2ab.jar
> 2014-08-25 07:07:49 b.s.d.nimbus [INFO] Received topology submission for
> aleads with conf {"topology.max.task.parallelism" nil,
> "topology.acker.executors" nil, "topology.kryo.register" nil,
> "topology.kryo.decorators" (), "topology.name" "aleads", "storm.id"
> "aleads-3-1408964869", "modelId" "ut", "topology.workers" 1,
> "topology.debug" true}
> 2014-08-25 07:07:50 b.s.d.nimbus [INFO] Activating aleads:
> aleads-3-1408964869
> 2014-08-25 07:07:50 b.s.s.EvenScheduler [INFO] Available slots:
> (["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701]
> ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6700])
> 2014-08-25 07:07:50 b.s.d.nimbus [INFO] Setting new assignment for
> topology id aleads-3-1408964869:
> #backtype.storm.daemon.common.Assignment{:master-code-dir
> "/hadoop/storm/nimbus/stormdist/aleads-3-1408964869", :node->host
> {"e56c2cc7-d35a-4355-9906-506618ff70c5" "hdp.ambari"}, :executor->node+port
> {[2 2] ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [3 3]
> ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [4 4]
> ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [5 5]
> ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [6 6]
> ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [7 7]
> ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [8 8]
> ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [9 9]
> ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701], [1 1]
> ["e56c2cc7-d35a-4355-9906-506618ff70c5" 6701]}, :executor->start-time-secs
> {[1 1] 1408964870, [9 9] 1408964870, [8 8] 1408964870, [7 7] 1408964870, [6
> 6] 1408964870, [5 5] 1408964870, [4 4] 1408964870, [3 3] 1408964870, [2 2]
> 1408964870}}
>
> Can anyone guess what I have done wrong and why Storm is not giving any
> error log anywhere.
>
> Storm version is 0.9.1.2.1.3.0-563 (Installed via HortonWorks)
> Kafka version is 2.10-0.8.1.1
> Storm-Kafka version 0.9.2-incubating
>
> --
> Regards,
> Vikas Agarwal
> 91 – 9928301411
>
> InfoObjects, Inc.
> Execution Matters
> http://www.infoobjects.com
> 2041 Mission College Boulevard, #280
> Santa Clara, CA 95054
> +1 (408) 988-2000 Work
> +1 (408) 716-2726 Fax
>
>
--
Regards,
Vikas Agarwal
91 – 9928301411
InfoObjects, Inc.
Execution Matters
http://www.infoobjects.com
2041 Mission College Boulevard, #280
Santa Clara, CA 95054
+1 (408) 988-2000 Work
+1 (408) 716-2726 Fax