You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Matthias J. Sax" <ma...@confluent.io> on 2018/09/14 18:44:11 UTC
Re: [DISCUSS] KIP-262 Metadata should include the number of state
stores for task
Hi,
we recently had a discussion on a different ticket to reduce the size of
the metadata we need to send:
https://issues.apache.org/jira/browse/KAFKA-7149
It seems, that we actually don't need to include the number of stores in
the metadata, but that we can compute the number of stores locally on
each instance.
With this insight, we should still try to exploit this knowledge during
task assignment, however, this would be an internal change that does not
require a KIP. Thus, I think that we can discard this KIP.
Thoughts?
-Matthias
On 6/10/18 5:20 PM, Matthias J. Sax wrote:
> Richard,
>
> KIP-268 got merged and thus this KIP is unblocked.
>
> I just re-read it and think it needs some updates with regard to the
> upgrade path (ie, you should mention why upgrading is covered).
>
> It would also be useful to discuss how the store information is used
> during assignment. Atm, the KIP only discussed that the information
> should be added, but this is only half of the story from my point of view.
>
>
> -Matthias
>
> On 3/22/18 9:15 PM, Matthias J. Sax wrote:
>> Hi Richard,
>>
>> with KIP-268 in place (should be accepted soon) the upgrade path is
>> covered. Thus, you can update your KIP accordingly, referring to KIP-268.
>>
>> Can you also update your KIP similar to KIP-268 to cover the old and new
>> metadata format?
>>
>> Thanks!
>>
>> -Matthias
>>
>>
>> On 2/24/18 4:07 PM, Richard Yu wrote:
>>> I didn't really get what "upgrade strategy" was at the time that Guozhang
>>> mentioned it, so I wrote the above dialogue from my first understanding. I
>>> changed it to "upgrade strategy must be provided". Currently, however, I do
>>> not have anything in mind to facilitate upgrading older Kafka brokers. If
>>> you have anything in mind, please let me know.
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Feb 24, 2018 at 3:58 PM, Matthias J. Sax <ma...@confluent.io>
>>> wrote:
>>>
>>>> Thanks a lot for this KIP.
>>>>
>>>> I am not sure what you mean by
>>>>
>>>>> which could potentially break older versions of Kafka brokers
>>>>
>>>> The metadata that is exchange, is not interpreted by the brokers. The
>>>> problem with upgrading the metadata format affect only Kafka Streams
>>>> instances.
>>>>
>>>> If we don't provide an upgrade strategy, changing the metadata format
>>>> required to stop all running application instances, before the instances
>>>> can be restarted with the new code. However, this implies downtime for
>>>> an application and is thus not acceptable.
>>>>
>>>>
>>>> -Matthias
>>>>
>>>>
>>>> On 2/24/18 11:11 AM, Richard Yu wrote:
>>>>> Hi all,
>>>>>
>>>>> I would like to discuss a KIP I've submitted :
>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>> 262%3A+Metadata+should+include+number+of+state+stores+for+task
>>>>>
>>>>>
>>>>> Regards,
>>>>> Richard Yu
>>>>>
>>>>
>>>>
>>>
>>
>
Re: [DISCUSS] KIP-262 Metadata should include the number of state
stores for task
Posted by "Matthias J. Sax" <ma...@confluent.io>.
This is still open.
If you agree that we don't need it any longer, please update the wiki
pages accordingly and move the KIP into table "Discarded KIPs":
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-DiscardedKIPs
-Mattias
On 1/3/19 10:41 PM, Richard Yu wrote:
> Hi Matthias,
>
> I guess this is no longer necessary. I am open to anything honestly.
> I suppose we should close it (if its not already).
>
>
> On Fri, Oct 19, 2018 at 11:06 AM Matthias J. Sax <ma...@confluent.io>
> wrote:
>
>> Any thought on my last email about discarding this KIP?
>>
>>
>> -Matthias
>>
>> On 9/14/18 11:44 AM, Matthias J. Sax wrote:
>>> Hi,
>>>
>>> we recently had a discussion on a different ticket to reduce the size of
>>> the metadata we need to send:
>>> https://issues.apache.org/jira/browse/KAFKA-7149
>>>
>>> It seems, that we actually don't need to include the number of stores in
>>> the metadata, but that we can compute the number of stores locally on
>>> each instance.
>>>
>>> With this insight, we should still try to exploit this knowledge during
>>> task assignment, however, this would be an internal change that does not
>>> require a KIP. Thus, I think that we can discard this KIP.
>>>
>>> Thoughts?
>>>
>>>
>>> -Matthias
>>>
>>> On 6/10/18 5:20 PM, Matthias J. Sax wrote:
>>>> Richard,
>>>>
>>>> KIP-268 got merged and thus this KIP is unblocked.
>>>>
>>>> I just re-read it and think it needs some updates with regard to the
>>>> upgrade path (ie, you should mention why upgrading is covered).
>>>>
>>>> It would also be useful to discuss how the store information is used
>>>> during assignment. Atm, the KIP only discussed that the information
>>>> should be added, but this is only half of the story from my point of
>> view.
>>>>
>>>>
>>>> -Matthias
>>>>
>>>> On 3/22/18 9:15 PM, Matthias J. Sax wrote:
>>>>> Hi Richard,
>>>>>
>>>>> with KIP-268 in place (should be accepted soon) the upgrade path is
>>>>> covered. Thus, you can update your KIP accordingly, referring to
>> KIP-268.
>>>>>
>>>>> Can you also update your KIP similar to KIP-268 to cover the old and
>> new
>>>>> metadata format?
>>>>>
>>>>> Thanks!
>>>>>
>>>>> -Matthias
>>>>>
>>>>>
>>>>> On 2/24/18 4:07 PM, Richard Yu wrote:
>>>>>> I didn't really get what "upgrade strategy" was at the time that
>> Guozhang
>>>>>> mentioned it, so I wrote the above dialogue from my first
>> understanding. I
>>>>>> changed it to "upgrade strategy must be provided". Currently,
>> however, I do
>>>>>> not have anything in mind to facilitate upgrading older Kafka
>> brokers. If
>>>>>> you have anything in mind, please let me know.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, Feb 24, 2018 at 3:58 PM, Matthias J. Sax <
>> matthias@confluent.io>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks a lot for this KIP.
>>>>>>>
>>>>>>> I am not sure what you mean by
>>>>>>>
>>>>>>>> which could potentially break older versions of Kafka brokers
>>>>>>>
>>>>>>> The metadata that is exchange, is not interpreted by the brokers. The
>>>>>>> problem with upgrading the metadata format affect only Kafka Streams
>>>>>>> instances.
>>>>>>>
>>>>>>> If we don't provide an upgrade strategy, changing the metadata format
>>>>>>> required to stop all running application instances, before the
>> instances
>>>>>>> can be restarted with the new code. However, this implies downtime
>> for
>>>>>>> an application and is thus not acceptable.
>>>>>>>
>>>>>>>
>>>>>>> -Matthias
>>>>>>>
>>>>>>>
>>>>>>> On 2/24/18 11:11 AM, Richard Yu wrote:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I would like to discuss a KIP I've submitted :
>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>> 262%3A+Metadata+should+include+number+of+state+stores+for+task
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Richard Yu
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>
Re: [DISCUSS] KIP-262 Metadata should include the number of state
stores for task
Posted by Richard Yu <yo...@gmail.com>.
Hi Matthias,
I guess this is no longer necessary. I am open to anything honestly.
I suppose we should close it (if its not already).
On Fri, Oct 19, 2018 at 11:06 AM Matthias J. Sax <ma...@confluent.io>
wrote:
> Any thought on my last email about discarding this KIP?
>
>
> -Matthias
>
> On 9/14/18 11:44 AM, Matthias J. Sax wrote:
> > Hi,
> >
> > we recently had a discussion on a different ticket to reduce the size of
> > the metadata we need to send:
> > https://issues.apache.org/jira/browse/KAFKA-7149
> >
> > It seems, that we actually don't need to include the number of stores in
> > the metadata, but that we can compute the number of stores locally on
> > each instance.
> >
> > With this insight, we should still try to exploit this knowledge during
> > task assignment, however, this would be an internal change that does not
> > require a KIP. Thus, I think that we can discard this KIP.
> >
> > Thoughts?
> >
> >
> > -Matthias
> >
> > On 6/10/18 5:20 PM, Matthias J. Sax wrote:
> >> Richard,
> >>
> >> KIP-268 got merged and thus this KIP is unblocked.
> >>
> >> I just re-read it and think it needs some updates with regard to the
> >> upgrade path (ie, you should mention why upgrading is covered).
> >>
> >> It would also be useful to discuss how the store information is used
> >> during assignment. Atm, the KIP only discussed that the information
> >> should be added, but this is only half of the story from my point of
> view.
> >>
> >>
> >> -Matthias
> >>
> >> On 3/22/18 9:15 PM, Matthias J. Sax wrote:
> >>> Hi Richard,
> >>>
> >>> with KIP-268 in place (should be accepted soon) the upgrade path is
> >>> covered. Thus, you can update your KIP accordingly, referring to
> KIP-268.
> >>>
> >>> Can you also update your KIP similar to KIP-268 to cover the old and
> new
> >>> metadata format?
> >>>
> >>> Thanks!
> >>>
> >>> -Matthias
> >>>
> >>>
> >>> On 2/24/18 4:07 PM, Richard Yu wrote:
> >>>> I didn't really get what "upgrade strategy" was at the time that
> Guozhang
> >>>> mentioned it, so I wrote the above dialogue from my first
> understanding. I
> >>>> changed it to "upgrade strategy must be provided". Currently,
> however, I do
> >>>> not have anything in mind to facilitate upgrading older Kafka
> brokers. If
> >>>> you have anything in mind, please let me know.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Sat, Feb 24, 2018 at 3:58 PM, Matthias J. Sax <
> matthias@confluent.io>
> >>>> wrote:
> >>>>
> >>>>> Thanks a lot for this KIP.
> >>>>>
> >>>>> I am not sure what you mean by
> >>>>>
> >>>>>> which could potentially break older versions of Kafka brokers
> >>>>>
> >>>>> The metadata that is exchange, is not interpreted by the brokers. The
> >>>>> problem with upgrading the metadata format affect only Kafka Streams
> >>>>> instances.
> >>>>>
> >>>>> If we don't provide an upgrade strategy, changing the metadata format
> >>>>> required to stop all running application instances, before the
> instances
> >>>>> can be restarted with the new code. However, this implies downtime
> for
> >>>>> an application and is thus not acceptable.
> >>>>>
> >>>>>
> >>>>> -Matthias
> >>>>>
> >>>>>
> >>>>> On 2/24/18 11:11 AM, Richard Yu wrote:
> >>>>>> Hi all,
> >>>>>>
> >>>>>> I would like to discuss a KIP I've submitted :
> >>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>>> 262%3A+Metadata+should+include+number+of+state+stores+for+task
> >>>>>>
> >>>>>>
> >>>>>> Regards,
> >>>>>> Richard Yu
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
>
>
Re: [DISCUSS] KIP-262 Metadata should include the number of state
stores for task
Posted by "Matthias J. Sax" <ma...@confluent.io>.
Any thought on my last email about discarding this KIP?
-Matthias
On 9/14/18 11:44 AM, Matthias J. Sax wrote:
> Hi,
>
> we recently had a discussion on a different ticket to reduce the size of
> the metadata we need to send:
> https://issues.apache.org/jira/browse/KAFKA-7149
>
> It seems, that we actually don't need to include the number of stores in
> the metadata, but that we can compute the number of stores locally on
> each instance.
>
> With this insight, we should still try to exploit this knowledge during
> task assignment, however, this would be an internal change that does not
> require a KIP. Thus, I think that we can discard this KIP.
>
> Thoughts?
>
>
> -Matthias
>
> On 6/10/18 5:20 PM, Matthias J. Sax wrote:
>> Richard,
>>
>> KIP-268 got merged and thus this KIP is unblocked.
>>
>> I just re-read it and think it needs some updates with regard to the
>> upgrade path (ie, you should mention why upgrading is covered).
>>
>> It would also be useful to discuss how the store information is used
>> during assignment. Atm, the KIP only discussed that the information
>> should be added, but this is only half of the story from my point of view.
>>
>>
>> -Matthias
>>
>> On 3/22/18 9:15 PM, Matthias J. Sax wrote:
>>> Hi Richard,
>>>
>>> with KIP-268 in place (should be accepted soon) the upgrade path is
>>> covered. Thus, you can update your KIP accordingly, referring to KIP-268.
>>>
>>> Can you also update your KIP similar to KIP-268 to cover the old and new
>>> metadata format?
>>>
>>> Thanks!
>>>
>>> -Matthias
>>>
>>>
>>> On 2/24/18 4:07 PM, Richard Yu wrote:
>>>> I didn't really get what "upgrade strategy" was at the time that Guozhang
>>>> mentioned it, so I wrote the above dialogue from my first understanding. I
>>>> changed it to "upgrade strategy must be provided". Currently, however, I do
>>>> not have anything in mind to facilitate upgrading older Kafka brokers. If
>>>> you have anything in mind, please let me know.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Sat, Feb 24, 2018 at 3:58 PM, Matthias J. Sax <ma...@confluent.io>
>>>> wrote:
>>>>
>>>>> Thanks a lot for this KIP.
>>>>>
>>>>> I am not sure what you mean by
>>>>>
>>>>>> which could potentially break older versions of Kafka brokers
>>>>>
>>>>> The metadata that is exchange, is not interpreted by the brokers. The
>>>>> problem with upgrading the metadata format affect only Kafka Streams
>>>>> instances.
>>>>>
>>>>> If we don't provide an upgrade strategy, changing the metadata format
>>>>> required to stop all running application instances, before the instances
>>>>> can be restarted with the new code. However, this implies downtime for
>>>>> an application and is thus not acceptable.
>>>>>
>>>>>
>>>>> -Matthias
>>>>>
>>>>>
>>>>> On 2/24/18 11:11 AM, Richard Yu wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> I would like to discuss a KIP I've submitted :
>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>> 262%3A+Metadata+should+include+number+of+state+stores+for+task
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Richard Yu
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>