You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by ChienHsing Wu <ch...@opentext.com> on 2018/10/30 14:56:18 UTC
RE: [EXTERNAL] - Re: KAFKA-3932 - Consumer fails to consume in a
round robin fashion
Hi Matthias,
Sorry about the late reply.
I have a Jira account. It's chienhsw. I am using the latest version 2.0. Would it be possible to add that to 2.0 as a minor release?
Thanks, ChienHsing
-----Original Message-----
From: Matthias J. Sax <ma...@confluent.io>
Sent: Wednesday, October 24, 2018 6:41 PM
To: dev@kafka.apache.org
Subject: [EXTERNAL] - Re: KAFKA-3932 - Consumer fails to consume in a round robin fashion
CH,
Thanks for contributing to Kafka. Do you have a Jira account already? If yes, what is your account id? If not, you need to create one first and share your id so we can grant permission to self-assign tickets.
I was just looking into the ticket itself, and it's marked as 0.10.0.0.
You say you encountered this issues. Do you use 0.10.0.x version? AFAIK, the consumer was updated in later versions, and the behavior should be different. Before you start working on the ticket, we should verify that it is not already fixed. For this case, we would just resolve the ticket with corresponding fixed version.
Note, that the behavior (at least from my point of view) is not a bug, but addressing it would be an improvement. Thus, if you work on it, the patch would be released with 2.2.0 version, but _not_ with a potential
0.10.0.2 release.
Does this make sense?
-Matthias
On 10/24/18 6:27 AM, ChienHsing Wu wrote:
> I don't see any comments/concerns. I would like to implement and commit to this ticket. Could anyone let me know how to request for the permission to assign that ticket to me?
>
> Thanks, CH
>
> From: ChienHsing Wu
> Sent: Monday, October 22, 2018 1:40 PM
> To: 'dev@kafka.apache.org' <de...@kafka.apache.org>
> Subject: KAFKA-3932 - Consumer fails to consume in a round robin
> fashion
>
>
> Hi,
>
>
>
> I encountered the issue documented in the jira KAFKA-3932<https://issues.apache.org/jira/browse/KAFKA-3932?jql=text%20~%20%22round%20robin%20consumer%22>. Upon studying the source code and the PIP<https://cwiki.apache.org/confluence/display/KAFKA/KIP-41%3A+KafkaConsumer+Max+Records>, I think the issues is the statement in PIP: "As before, we'd keep track of which partition we left off at so that the next iteration would begin there." I think it should NOT use the last partition in the next iteration; it should pick the next one instead.
>
> If this behavior is agreeable, the simplest solution to impose the order to pick the next one is to use the order the consumer.internals.Fetcher receives the partition messages, as determined by completedFetches queue in that class. To avoid parsing the partition messages repeatedly. we can save those parsed fetches to a list and maintain the next partition to get messages there.
>
> Does it sound like a good approach? If this is not the right place to discuss the design please let me know where to engage. If this is agreeable I can contribute the implementation.
>
>
>
> Thanks, CH
>
>
RE: [EXTERNAL] - Re: KAFKA-3932 - Consumer fails to consume in a
round robin fashion
Posted by ChienHsing Wu <ch...@opentext.com>.
Thanks Matthias. I just assigned that ticket to myself.
This ticket is about what returns from the poll call from the client's perspective. I will ask the reporter to confirm.
I will request to enter a KIP.
CH
-----Original Message-----
From: Matthias J. Sax <ma...@confluent.io>
Sent: Tuesday, October 30, 2018 2:44 PM
To: dev@kafka.apache.org
Subject: Re: [EXTERNAL] - Re: KAFKA-3932 - Consumer fails to consume in a round robin fashion
I added you to the list of contributes. You can now self-assign ticket to yourself.
Before you start working on this, we need to understand what the actual issue is in detail. Note, that sending fetch request to partitions and what poll() returns is two different things by design.
You might also want to read
https://cwiki.apache.org/confluence/display/KAFKA/KIP-227%3A+Introduce+Incremental+FetchRequests+to+Increase+Partition+Scalability
I am not sure, if KAFKA-3923 requests to change the fetch request logic (what was done already) or what poll() returns from the buffer, or maybe both. It would be good to ask for clarification on the ticket to see what the reported actually means, and if the current behavior meets there requirements and if not, why.
Overall, this change seems to require a KIP anyway. Hope this helps.
-Matthias
On 10/30/18 9:38 AM, ChienHsing Wu wrote:
> I just looked at the release schedule. I guess the 2.2 is around
> Feb/2019, right? --CH
>
> -----Original Message-----
> From: ChienHsing Wu <ch...@opentext.com>
> Sent: Tuesday, October 30, 2018 10:56 AM
> To: dev@kafka.apache.org
> Subject: RE: [EXTERNAL] - Re: KAFKA-3932 - Consumer fails to consume
> in a round robin fashion
>
> Hi Matthias,
>
> Sorry about the late reply.
>
> I have a Jira account. It's chienhsw. I am using the latest version 2.0. Would it be possible to add that to 2.0 as a minor release?
>
> Thanks, ChienHsing
>
> -----Original Message-----
> From: Matthias J. Sax <ma...@confluent.io>
> Sent: Wednesday, October 24, 2018 6:41 PM
> To: dev@kafka.apache.org
> Subject: [EXTERNAL] - Re: KAFKA-3932 - Consumer fails to consume in a
> round robin fashion
>
> CH,
>
> Thanks for contributing to Kafka. Do you have a Jira account already? If yes, what is your account id? If not, you need to create one first and share your id so we can grant permission to self-assign tickets.
>
> I was just looking into the ticket itself, and it's marked as 0.10.0.0.
> You say you encountered this issues. Do you use 0.10.0.x version? AFAIK, the consumer was updated in later versions, and the behavior should be different. Before you start working on the ticket, we should verify that it is not already fixed. For this case, we would just resolve the ticket with corresponding fixed version.
>
> Note, that the behavior (at least from my point of view) is not a bug,
> but addressing it would be an improvement. Thus, if you work on it,
> the patch would be released with 2.2.0 version, but _not_ with a
> potential
> 0.10.0.2 release.
>
> Does this make sense?
>
>
> -Matthias
>
> On 10/24/18 6:27 AM, ChienHsing Wu wrote:
>> I don't see any comments/concerns. I would like to implement and commit to this ticket. Could anyone let me know how to request for the permission to assign that ticket to me?
>>
>> Thanks, CH
>>
>> From: ChienHsing Wu
>> Sent: Monday, October 22, 2018 1:40 PM
>> To: 'dev@kafka.apache.org' <de...@kafka.apache.org>
>> Subject: KAFKA-3932 - Consumer fails to consume in a round robin
>> fashion
>>
>>
>> Hi,
>>
>>
>>
>> I encountered the issue documented in the jira KAFKA-3932<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_KAFKA-2D3932-3Fjql-3Dtext-2520-7E-2520-2522round-2520robin-2520consumer-2522&d=DwIFAg&c=ZgVRmm3mf2P1-XDAyDsu4A&r=Az03wMrbL9ToLW0OFyo3wo3985rhAKPMLmmg6RA3V7I&m=HLTuFMPfB0s5o_WTleQx1c5YRdKxDWwoJRC4iDkPopw&s=9KsjjzejGA5jiySmpE3WR0wqAoOKfZhju-8dUhZZoD4&e=>. Upon studying the source code and the PIP<https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D41-253A-2BKafkaConsumer-2BMax-2BRecords&d=DwIFAg&c=ZgVRmm3mf2P1-XDAyDsu4A&r=Az03wMrbL9ToLW0OFyo3wo3985rhAKPMLmmg6RA3V7I&m=HLTuFMPfB0s5o_WTleQx1c5YRdKxDWwoJRC4iDkPopw&s=L7JVo7fEjkgHrA0jN5SUmCIrxlQuIRWCxG_iiBD-zdw&e=>, I think the issues is the statement in PIP: "As before, we'd keep track of which partition we left off at so that the next iteration would begin there." I think it should NOT use the last partition in the next iteration; it should pick the next one instead.
>>
>> If this behavior is agreeable, the simplest solution to impose the order to pick the next one is to use the order the consumer.internals.Fetcher receives the partition messages, as determined by completedFetches queue in that class. To avoid parsing the partition messages repeatedly. we can save those parsed fetches to a list and maintain the next partition to get messages there.
>>
>> Does it sound like a good approach? If this is not the right place to discuss the design please let me know where to engage. If this is agreeable I can contribute the implementation.
>>
>>
>>
>> Thanks, CH
>>
>>
>
Re: [EXTERNAL] - Re: KAFKA-3932 - Consumer fails to consume in a
round robin fashion
Posted by "Matthias J. Sax" <ma...@confluent.io>.
I added you to the list of contributes. You can now self-assign ticket
to yourself.
Before you start working on this, we need to understand what the actual
issue is in detail. Note, that sending fetch request to partitions and
what poll() returns is two different things by design.
You might also want to read
https://cwiki.apache.org/confluence/display/KAFKA/KIP-227%3A+Introduce+Incremental+FetchRequests+to+Increase+Partition+Scalability
I am not sure, if KAFKA-3923 requests to change the fetch request logic
(what was done already) or what poll() returns from the buffer, or maybe
both. It would be good to ask for clarification on the ticket to see
what the reported actually means, and if the current behavior meets
there requirements and if not, why.
Overall, this change seems to require a KIP anyway. Hope this helps.
-Matthias
On 10/30/18 9:38 AM, ChienHsing Wu wrote:
> I just looked at the release schedule. I guess the 2.2 is around Feb/2019, right? --CH
>
> -----Original Message-----
> From: ChienHsing Wu <ch...@opentext.com>
> Sent: Tuesday, October 30, 2018 10:56 AM
> To: dev@kafka.apache.org
> Subject: RE: [EXTERNAL] - Re: KAFKA-3932 - Consumer fails to consume in a round robin fashion
>
> Hi Matthias,
>
> Sorry about the late reply.
>
> I have a Jira account. It's chienhsw. I am using the latest version 2.0. Would it be possible to add that to 2.0 as a minor release?
>
> Thanks, ChienHsing
>
> -----Original Message-----
> From: Matthias J. Sax <ma...@confluent.io>
> Sent: Wednesday, October 24, 2018 6:41 PM
> To: dev@kafka.apache.org
> Subject: [EXTERNAL] - Re: KAFKA-3932 - Consumer fails to consume in a round robin fashion
>
> CH,
>
> Thanks for contributing to Kafka. Do you have a Jira account already? If yes, what is your account id? If not, you need to create one first and share your id so we can grant permission to self-assign tickets.
>
> I was just looking into the ticket itself, and it's marked as 0.10.0.0.
> You say you encountered this issues. Do you use 0.10.0.x version? AFAIK, the consumer was updated in later versions, and the behavior should be different. Before you start working on the ticket, we should verify that it is not already fixed. For this case, we would just resolve the ticket with corresponding fixed version.
>
> Note, that the behavior (at least from my point of view) is not a bug, but addressing it would be an improvement. Thus, if you work on it, the patch would be released with 2.2.0 version, but _not_ with a potential
> 0.10.0.2 release.
>
> Does this make sense?
>
>
> -Matthias
>
> On 10/24/18 6:27 AM, ChienHsing Wu wrote:
>> I don't see any comments/concerns. I would like to implement and commit to this ticket. Could anyone let me know how to request for the permission to assign that ticket to me?
>>
>> Thanks, CH
>>
>> From: ChienHsing Wu
>> Sent: Monday, October 22, 2018 1:40 PM
>> To: 'dev@kafka.apache.org' <de...@kafka.apache.org>
>> Subject: KAFKA-3932 - Consumer fails to consume in a round robin
>> fashion
>>
>>
>> Hi,
>>
>>
>>
>> I encountered the issue documented in the jira KAFKA-3932<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_KAFKA-2D3932-3Fjql-3Dtext-2520-7E-2520-2522round-2520robin-2520consumer-2522&d=DwIFAg&c=ZgVRmm3mf2P1-XDAyDsu4A&r=Az03wMrbL9ToLW0OFyo3wo3985rhAKPMLmmg6RA3V7I&m=HLTuFMPfB0s5o_WTleQx1c5YRdKxDWwoJRC4iDkPopw&s=9KsjjzejGA5jiySmpE3WR0wqAoOKfZhju-8dUhZZoD4&e=>. Upon studying the source code and the PIP<https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D41-253A-2BKafkaConsumer-2BMax-2BRecords&d=DwIFAg&c=ZgVRmm3mf2P1-XDAyDsu4A&r=Az03wMrbL9ToLW0OFyo3wo3985rhAKPMLmmg6RA3V7I&m=HLTuFMPfB0s5o_WTleQx1c5YRdKxDWwoJRC4iDkPopw&s=L7JVo7fEjkgHrA0jN5SUmCIrxlQuIRWCxG_iiBD-zdw&e=>, I think the issues is the statement in PIP: "As before, we'd keep track of which partition we left off at so that the next iteration would begin there." I think it should NOT use the last partition in the next iteration; it should pick the next one instead.
>>
>> If this behavior is agreeable, the simplest solution to impose the order to pick the next one is to use the order the consumer.internals.Fetcher receives the partition messages, as determined by completedFetches queue in that class. To avoid parsing the partition messages repeatedly. we can save those parsed fetches to a list and maintain the next partition to get messages there.
>>
>> Does it sound like a good approach? If this is not the right place to discuss the design please let me know where to engage. If this is agreeable I can contribute the implementation.
>>
>>
>>
>> Thanks, CH
>>
>>
>
RE: [EXTERNAL] - Re: KAFKA-3932 - Consumer fails to consume in a
round robin fashion
Posted by ChienHsing Wu <ch...@opentext.com>.
I just looked at the release schedule. I guess the 2.2 is around Feb/2019, right? --CH
-----Original Message-----
From: ChienHsing Wu <ch...@opentext.com>
Sent: Tuesday, October 30, 2018 10:56 AM
To: dev@kafka.apache.org
Subject: RE: [EXTERNAL] - Re: KAFKA-3932 - Consumer fails to consume in a round robin fashion
Hi Matthias,
Sorry about the late reply.
I have a Jira account. It's chienhsw. I am using the latest version 2.0. Would it be possible to add that to 2.0 as a minor release?
Thanks, ChienHsing
-----Original Message-----
From: Matthias J. Sax <ma...@confluent.io>
Sent: Wednesday, October 24, 2018 6:41 PM
To: dev@kafka.apache.org
Subject: [EXTERNAL] - Re: KAFKA-3932 - Consumer fails to consume in a round robin fashion
CH,
Thanks for contributing to Kafka. Do you have a Jira account already? If yes, what is your account id? If not, you need to create one first and share your id so we can grant permission to self-assign tickets.
I was just looking into the ticket itself, and it's marked as 0.10.0.0.
You say you encountered this issues. Do you use 0.10.0.x version? AFAIK, the consumer was updated in later versions, and the behavior should be different. Before you start working on the ticket, we should verify that it is not already fixed. For this case, we would just resolve the ticket with corresponding fixed version.
Note, that the behavior (at least from my point of view) is not a bug, but addressing it would be an improvement. Thus, if you work on it, the patch would be released with 2.2.0 version, but _not_ with a potential
0.10.0.2 release.
Does this make sense?
-Matthias
On 10/24/18 6:27 AM, ChienHsing Wu wrote:
> I don't see any comments/concerns. I would like to implement and commit to this ticket. Could anyone let me know how to request for the permission to assign that ticket to me?
>
> Thanks, CH
>
> From: ChienHsing Wu
> Sent: Monday, October 22, 2018 1:40 PM
> To: 'dev@kafka.apache.org' <de...@kafka.apache.org>
> Subject: KAFKA-3932 - Consumer fails to consume in a round robin
> fashion
>
>
> Hi,
>
>
>
> I encountered the issue documented in the jira KAFKA-3932<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_KAFKA-2D3932-3Fjql-3Dtext-2520-7E-2520-2522round-2520robin-2520consumer-2522&d=DwIFAg&c=ZgVRmm3mf2P1-XDAyDsu4A&r=Az03wMrbL9ToLW0OFyo3wo3985rhAKPMLmmg6RA3V7I&m=HLTuFMPfB0s5o_WTleQx1c5YRdKxDWwoJRC4iDkPopw&s=9KsjjzejGA5jiySmpE3WR0wqAoOKfZhju-8dUhZZoD4&e=>. Upon studying the source code and the PIP<https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D41-253A-2BKafkaConsumer-2BMax-2BRecords&d=DwIFAg&c=ZgVRmm3mf2P1-XDAyDsu4A&r=Az03wMrbL9ToLW0OFyo3wo3985rhAKPMLmmg6RA3V7I&m=HLTuFMPfB0s5o_WTleQx1c5YRdKxDWwoJRC4iDkPopw&s=L7JVo7fEjkgHrA0jN5SUmCIrxlQuIRWCxG_iiBD-zdw&e=>, I think the issues is the statement in PIP: "As before, we'd keep track of which partition we left off at so that the next iteration would begin there." I think it should NOT use the last partition in the next iteration; it should pick the next one instead.
>
> If this behavior is agreeable, the simplest solution to impose the order to pick the next one is to use the order the consumer.internals.Fetcher receives the partition messages, as determined by completedFetches queue in that class. To avoid parsing the partition messages repeatedly. we can save those parsed fetches to a list and maintain the next partition to get messages there.
>
> Does it sound like a good approach? If this is not the right place to discuss the design please let me know where to engage. If this is agreeable I can contribute the implementation.
>
>
>
> Thanks, CH
>
>