Regarding Kafka message pushing

[SAURABH VERMA <sa...@gmail.com>]

Hello,

  I am experimenting with Kafka. Let me explain in brief what I am trying
to do. I am listening messages from a very high-speed velocity bus,
processing them and pushing to a Kafka cluster. The Kafka cluster has 5
broker nodes (and a second setup has 10 broker nodes). I am facing an
always reproducible issue on both the setups that after throwing 66K
messages to any of the above configurations, I am getting a
FailedToSendMessage exception at the Kafka producers that are pushing the
processed messages to the Kafka cluster. I am unable to figure out like
what is the issue. One more info., if I batch 1000 messages in a single
push, then also the issue occurs after 66K throws. Thus, size of the push
is not playing a role here but the no. of pushes.

One more thing, when I set producer.type="async", the producers throw OOME
as soon as I run them.

Kindly let me know if you have faced similar issues or some point that I
may be missing.

I am unable to access http://chat.freenode.net/ as mentioned in Kafka
documentation.

-- 
Thanks & Regards,
Saurabh Verma,
India

Re: Regarding Kafka message pushing

[Jun Rao <ju...@gmail.com>]

Which version of Kafka are you using? The log before FailedToSendMessage
should tell you the cause. Could you paste that?

Thanks,

Jun


On Sun, May 25, 2014 at 10:59 PM, SAURABH VERMA <sa...@gmail.com>wrote:

> Hello,
>
>   I am experimenting with Kafka. Let me explain in brief what I am trying
> to do. I am listening messages from a very high-speed velocity bus,
> processing them and pushing to a Kafka cluster. The Kafka cluster has 5
> broker nodes (and a second setup has 10 broker nodes). I am facing an
> always reproducible issue on both the setups that after throwing 66K
> messages to any of the above configurations, I am getting a
> FailedToSendMessage exception at the Kafka producers that are pushing the
> processed messages to the Kafka cluster. I am unable to figure out like
> what is the issue. One more info., if I batch 1000 messages in a single
> push, then also the issue occurs after 66K throws. Thus, size of the push
> is not playing a role here but the no. of pushes.
>
> One more thing, when I set producer.type="async", the producers throw OOME
> as soon as I run them.
>
> Kindly let me know if you have faced similar issues or some point that I
> may be missing.
>
> I am unable to access http://chat.freenode.net/ as mentioned in Kafka
> documentation.
>
> --
> Thanks & Regards,
> Saurabh Verma,
> India
>