Strange 'emitted' numbers behavior in Topology stats (Storm 1.0.3)

[Dmitry Semenov <dm...@saritasa.com>]

hi Guys,

this is what my stat looks like
http://pasteboard.co/39qZsVjX2.png

The problem is that I have no idea where those numbers (of emitted tuples
are coming from).

   - My topology is pretty simple: kafka spout -> bolt (persisting data
   into hbase)
   - topology works - when I put data into kafka topic, I get them
   processed by bolt and persisted in hbase, which I then verify with scan
   operator in hbase shell (so new records are being inserted)
   - however each time I submit new message into kafka and when it’s
   persisted by bolt - my topology doesn’t increase number of emitted by ‘1’.
   - periodically I get all numbers increased by 20 - without sending any
   new messages into kafka. I.e. my kafka topic gets no messages for hours,
   but the number of tuples emitted always get increased in chunks of 20 over
   time. I still get the same number of records in hbase.
   - I get no exceptions/errors anywhere in apache storm logs.
   - I’m not doing ack() or fail() any of my tuples in bolt implementation
   - my capacity or latency in bolt metrics is always staying zero even
   when I load a lot of messages in Kafka
   - my kafka offset log ($KAFKA/kafka-run-class.sh
   kafka.tools.ConsumerOffsetChecker) shows all the messages are processed and
   Kafka Lag for given topic/group is 0.

So my question:

   - what are those ‘stealth’ tuples that increase ‘emitted’ in both Spout
   and Bolt?
   - is it possible to enable ‘debugging’ in storm UI to see what those
   tuples are?
   - why capacity/latency in bolt metrics is always zero?

Environment details

I’m using Java 8 + Apache Storm 1.0.3

[devops@storm-wk1-prod]~/storm/supervisor/stormdist% storm version
Running: /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -client
-Ddaemon.name= -Dstorm.options= -Dstorm.home=/opt/apache-storm-1.0.3
-Dstorm.log.dir=/opt/apache-storm-1.0.3/logs
-Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
-Dstorm.conf.file= -cp
/opt/apache-storm-1.0.3/lib/storm-core-1.0.3.jar:/opt/apache-storm-1.0.3/lib/kryo-3.0.3.jar:/opt/apache-storm-1.0.3/lib/reflectasm-1.10.1.jar:/opt/apache-storm-1.0.3/lib/asm-5.0.3.jar:/opt/apache-storm-1.0.3/lib/minlog-1.3.0.jar:/opt/apache-storm-1.0.3/lib/objenesis-2.1.jar:/opt/apache-storm-1.0.3/lib/clojure-1.7.0.jar:/opt/apache-storm-1.0.3/lib/disruptor-3.3.2.jar:/opt/apache-storm-1.0.3/lib/log4j-api-2.1.jar:/opt/apache-storm-1.0.3/lib/log4j-core-2.1.jar:/opt/apache-storm-1.0.3/lib/log4j-slf4j-impl-2.1.jar:/opt/apache-storm-1.0.3/lib/slf4j-api-1.7.7.jar:/opt/apache-storm-1.0.3/lib/log4j-over-slf4j-1.6.6.jar:/opt/apache-storm-1.0.3/lib/servlet-api-2.5.jar:/opt/apache-storm-1.0.3/lib/storm-rename-hack-1.0.3.jar:/opt/storm/conf
org.apache.storm.utils.VersionInfo
Storm 1.0.3
URL https://git-wip-us.apache.org/repos/asf/storm.git -r
eac433b0beb3798c4723deb39b3c4fad446378f4
Branch (no branch)
Compiled by ptgoetz on 2017-02-07T20:22Z
From source with checksum c78e52de4b8a22d99551d45dfe9c1a4b

My *storm.yaml*:

storm.zookeeper.servers:
  - "10.138.0.8"
  - "10.138.0.9"
  - "10.138.0.16"

storm.zookeeper.port: 2181

nimbus.seeds: ["10.138.0.10"]

storm.local.dir: "/var/log/storm"
supervisor.slots.ports:
  - 6700
  - 6701
  - 6702
  - 6703

worker.childopts: "-Xmx768m"
nimbus.childopts: "-Xmx512m"
supervisor.childopts: "-Xmx256m"

thanks,
Dmitry
​