You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Vinayak Sharma <vi...@amagi.com> on 2017/05/26 12:58:46 UTC
Kafka Loss of messages
Hi,
I am experiencing loss of messages while scaling brokers up and down in
kafka cluster.
I came across this jira issue(link
<https://issues.apache.org/jira/browse/KAFKA-1561>). Can you confirm if
this is actually an issue in kafka or can the problem be solved by changing
some configuration parameters.
I am enclosing the server.properties that I am using.
Please get to back to me.
Regards,
Vinayak.
--
<http://www.amagi.com>
Re: Kafka Loss of messages
Posted by Vinayak Sharma <vi...@amagi.com>.
Hi Jun,
Please ignore the earlier mail.
I have a Kafka Cluster running on AWS machines i.e. 3 Zookeeper and 2
Brokers. I am trying to scale the brokers while producing and consuming
messages.
I am sending 1000 messages by batch producer every second for 200 seconds
for the following topics:
topics : Topic1(20 partitions) Topic2(40 partitions)
Replication Factor.
Add 1 broker:
Produced Consumed
Topic1 200000 199950
Topic2 200000 199975
Remove 1 broker:
Produced Consumed
Topic1 200000 199950
Topic2 200000 199825
I run a script which reassigns the partitions. And as it can be seen from
the above numbers that we see data loss at the consumer end.
Let me know if you need any further details.
Regards,
Vinayak
On 29 May 2017 at 13:03, Sanket Dhopeshwarkar <sa...@amagi.com> wrote:
> +Vinayak
>
> On 26 May 2017 at 21:44, Jun Rao <ju...@confluent.io> wrote:
>
>> Hi, Vinayak,
>>
>> I am not sure that KAFKA-1561 as reported is an issue. Could you describe
>> the scenario in your test in that jira in a bit more detail?
>>
>> Thanks,
>>
>> Jun
>>
>> On Fri, May 26, 2017 at 5:58 AM, Vinayak Sharma <vi...@amagi.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am experiencing loss of messages while scaling brokers up and down in
>>> kafka cluster.
>>>
>>> I came across this jira issue(link
>>> <https://issues.apache.org/jira/browse/KAFKA-1561>). Can you confirm if
>>> this is actually an issue in kafka or can the problem be solved by changing
>>> some configuration parameters.
>>>
>>> I am enclosing the server.properties that I am using.
>>>
>>> Please get to back to me.
>>>
>>> Regards,
>>> Vinayak.
>>>
>>> <http://www.amagi.com>
>>
>>
>>
>
--
<http://www.amagi.com>
Re: Kafka Loss of messages
Posted by Vinayak Sharma <vi...@amagi.com>.
Hi Jun,
0.10.0.0
I have a Kafka Cluster running on AWS instances. I am trying to scale
broker up and down while simultaneously using the cluster to produce and
consume messages.
On 29 May 2017 at 13:03, Sanket Dhopeshwarkar <sa...@amagi.com> wrote:
> +Vinayak
>
> On 26 May 2017 at 21:44, Jun Rao <ju...@confluent.io> wrote:
>
>> Hi, Vinayak,
>>
>> I am not sure that KAFKA-1561 as reported is an issue. Could you describe
>> the scenario in your test in that jira in a bit more detail?
>>
>> Thanks,
>>
>> Jun
>>
>> On Fri, May 26, 2017 at 5:58 AM, Vinayak Sharma <vi...@amagi.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am experiencing loss of messages while scaling brokers up and down in
>>> kafka cluster.
>>>
>>> I came across this jira issue(link
>>> <https://issues.apache.org/jira/browse/KAFKA-1561>). Can you confirm if
>>> this is actually an issue in kafka or can the problem be solved by changing
>>> some configuration parameters.
>>>
>>> I am enclosing the server.properties that I am using.
>>>
>>> Please get to back to me.
>>>
>>> Regards,
>>> Vinayak.
>>>
>>> <http://www.amagi.com>
>>
>>
>>
>
--
<http://www.amagi.com>
Re: Kafka Loss of messages
Posted by Sanket Dhopeshwarkar <sa...@amagi.com>.
+Vinayak
On 26 May 2017 at 21:44, Jun Rao <ju...@confluent.io> wrote:
> Hi, Vinayak,
>
> I am not sure that KAFKA-1561 as reported is an issue. Could you describe
> the scenario in your test in that jira in a bit more detail?
>
> Thanks,
>
> Jun
>
> On Fri, May 26, 2017 at 5:58 AM, Vinayak Sharma <vi...@amagi.com>
> wrote:
>
>> Hi,
>>
>> I am experiencing loss of messages while scaling brokers up and down in
>> kafka cluster.
>>
>> I came across this jira issue(link
>> <https://issues.apache.org/jira/browse/KAFKA-1561>). Can you confirm if
>> this is actually an issue in kafka or can the problem be solved by changing
>> some configuration parameters.
>>
>> I am enclosing the server.properties that I am using.
>>
>> Please get to back to me.
>>
>> Regards,
>> Vinayak.
>>
>> <http://www.amagi.com>
>
>
>
--
<http://www.amagi.com>
Re: Kafka Loss of messages
Posted by Jun Rao <ju...@confluent.io>.
Hi, Vinayak,
I am not sure that KAFKA-1561 as reported is an issue. Could you describe
the scenario in your test in that jira in a bit more detail?
Thanks,
Jun
On Fri, May 26, 2017 at 5:58 AM, Vinayak Sharma <vi...@amagi.com> wrote:
> Hi,
>
> I am experiencing loss of messages while scaling brokers up and down in
> kafka cluster.
>
> I came across this jira issue(link
> <https://issues.apache.org/jira/browse/KAFKA-1561>). Can you confirm if
> this is actually an issue in kafka or can the problem be solved by changing
> some configuration parameters.
>
> I am enclosing the server.properties that I am using.
>
> Please get to back to me.
>
> Regards,
> Vinayak.
>
> <http://www.amagi.com>