Number of Events Mismatch from source to destination

[Vineet Mishra <cl...@gmail.com>]

Hi All,

I am having a Kafka Storm Topology which is ingesting events published to
Kafka and processing on top of that data.

Although apart from some latency I found that everything was going good.
But recently I came across a issue which I couldn't get any solution yet.

I publishing some events from Logstash to Kafka and which is being
Subscribed by Storm Topology for further processing, I could see that the
source record count and the events processed by Storm is varying by a
reasonable number. So I have around 200 Million events to be processed out
of which 10 Million Events are getting lost as I could see the
acknowledgement of 190 Million events in the Storm.

Stuck at this issue, looking for expert advise.

Thanks!

Re: Number of Events Mismatch from source to destination

[Vineet Mishra <cl...@gmail.com>]

Hi Harsha,

The Log which I am processing is containing ~220 Million lines which are
transformed as events to Kafka, which indeed is getting processed by Storm.

The numbers while running Storm topology which I could see says,

Id
Executors
Tasks
Emitted
Transferred
Complete latency (ms)
Acked
Failed
Error Host
Error Port
Last error
KafkaSpout
<http://localhost:8080/component.html?id=KafkaSpout&topology_id=KafkaConsumerTopologyy-19-1423568605>
882179187402179187403.4942179377800 <http://:8000/log?file=worker-.log>

Which indeed is little less(~10 Million) than the total number of events,
hence I expect the Emitted Counts to be around ~220 Million which is
currently getting around ~217 Million.

Could you brief me through the nature of processing or why is the number
less than the expected count!

Thanks!

On Sun, Feb 15, 2015 at 9:53 PM, Harsha <st...@harsha.io> wrote:

>  Vineet,
>         How are you looking at number of events in kafka. Did you checked
> storm worker logs for any errors and what you mean by "the acknowledgement
> of 190 million events in storm" are you looking at number of acked messages?
> -Harsha
>
>
> On Sun, Feb 15, 2015, at 04:40 AM, Vineet Mishra wrote:
>
> Hi All,
>
> I am having a Kafka Storm Topology which is ingesting events published to
> Kafka and processing on top of that data.
>
> Although apart from some latency I found that everything was going good.
> But recently I came across a issue which I couldn't get any solution yet.
>
> I publishing some events from Logstash to Kafka and which is being
> Subscribed by Storm Topology for further processing, I could see that the
> source record count and the events processed by Storm is varying by a
> reasonable number. So I have around 200 Million events to be processed out
> of which 10 Million Events are getting lost as I could see the
> acknowledgement of 190 Million events in the Storm.
>
> Stuck at this issue, looking for expert advise.
>
> Thanks!
>
>
>

Re: Number of Events Mismatch from source to destination

[Harsha <st...@harsha.io>]

Vineet, How are you looking at number of events in kafka. Did you
checked storm worker logs for any errors and what you mean by "the
acknowledgement of 190 million events in storm" are you looking at
number of acked messages? -Harsha


On Sun, Feb 15, 2015, at 04:40 AM, Vineet Mishra wrote:
> Hi All,
>
> I am having a Kafka Storm Topology which is ingesting events published
> to Kafka and processing on top of that data.
>
> Although apart from some latency I found that everything was going
> good. But recently I came across a issue which I couldn't get any
> solution yet.
>
> I publishing some events from Logstash to Kafka and which is being
> Subscribed by Storm Topology for further processing, I could see that
> the source record count and the events processed by Storm is varying
> by a reasonable number. So I have around 200 Million events to be
> processed out of which 10 Million Events are getting lost as I could
> see the acknowledgement of 190 Million events in the Storm.
>
> Stuck at this issue, looking for expert advise.
>
> Thanks!