Samza stops processing

[Shekar Tippur <ct...@gmail.com>]

I have a strange issue. We just replaced  failed zookeeper server. We
killed the job in the interim and resubmitted the job to yarn.
The events were getting processed but at some point, the events stopped
processing. I have checked RM and application logs but they seem to be fine.

We then restarted Kafka and it seem to have fixed the issue. I have checked
into Kafka logs. Apart from some warn messages, there were no big red flags
either.
We are using Kafka 0.8 and Samza 0.8.

One of the things I did to test was to use the console producer and
produced a raw message. I am sure this could not have caused any issues.
 I am not sure how to debug this issue.

Appreciate any pointers.

Thanks,
Shekar

Re: Samza stops processing

[Jagadish Venkatraman <ja...@gmail.com>]

Hi Shekar,

When you say 'stopped' do you mean the container is is running and is
stalled/ hanged? Or do you mean the job failed?

Were you able to use the Kafka console consumer to consume  messages from
the same topics that your job is also consuming from?
ie, does the topic have 'upcoming' messages?








On Thu, Jun 9, 2016 at 1:40 AM, Shekar Tippur <ct...@gmail.com> wrote:

> I have a strange issue. We just replaced  failed zookeeper server. We
> killed the job in the interim and resubmitted the job to yarn.
> The events were getting processed but at some point, the events stopped
> processing. I have checked RM and application logs but they seem to be
> fine.
>
> We then restarted Kafka and it seem to have fixed the issue. I have checked
> into Kafka logs. Apart from some warn messages, there were no big red flags
> either.
> We are using Kafka 0.8 and Samza 0.8.
>
> One of the things I did to test was to use the console producer and
> produced a raw message. I am sure this could not have caused any issues.
>  I am not sure how to debug this issue.
>
> Appreciate any pointers.
>
> Thanks,
> Shekar
>



-- 
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University